Job Description
Role- PySpark Developer
nYear of Experience- 4 to 15 years
nLocation -Pune, Hyderabad, Chennai, Bangalore
nTechnical Skills:
n
n
Pysparkn
Python concepts and Frameworkn
Spark Architecturen
Big Datan
SQLn

nJob Description:
nJob Requirements*
n
n
Valuable work experience on Big Data Platforms like Hadoop, PySpark, Scala, Hive, Impala, SQL, Pythonn
Good Python, Pyspark, Big Data experiencen
Spark UI/Optimization/debugging techniquesn
Good python scripting skillsn
Intermediate SQL exposure – Subquery, Joins, CTE’sn
Database technologiesn

n

n
n Key Responsibilities
n
n
*Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processingn
.Architect end-to-end data validation systems in Hadoop setting for lineage, schema evolution
nLead system design for Hadoop/Hive test environments, including YARN resource management, energetic partitioningn




.Exposure to Zephyr-Jira-ServiceNow integrated test management systems with experience on API-driven automationn
.Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deploymentsn
.Create data quality system designs using PySpark integrated with Hive metadata servicesn
.Design testing platforms, test data generatorn
sMentor juniors on PySpark testing basics, contribute to testing strategy discussionn
sSpark session configurations for memory and core allocations for both local and cluster manager settingn
sData handling with distributed file systems like HDFS and writing back to hive tablen
sImplementation of Partitioning, caching techniques in organizing code for transformation pipelinen
sPerformance tuning implementation like salting, minimizing shufflinn

n

g

📌 Pyspark Developer Tamil Nadu
🏢 Tata Consultancy Services
📍 Tamil Nadu