Job Description
Role- PySpark Developer
nYear of Experience- 4 to 15 years
nLocation -Pune, Hyderabad, Chennai, Bangalore
nTechnical Skills:
n
- n
- Pysparkn
- Python concepts and Frameworkn
- Spark Architecturen
- Big Datan
- SQLn

nJob Description:
nJob Requirements*
n
- n
- Good work experience on Big Data Platforms like Hadoop, PySpark, Scala, Hive, Impala, SQL, Pythonn
- Good Python, Pyspark, Big Data experiencen
- Spark UI/Optimization/debugging techniquesn
- Good python scripting skillsn
- Intermediate SQL exposure – Subquery, Joins, CTE’sn
- Database technologiesn

n

n
n Key Responsibilities
n
- n
- *Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processingn
- .Architect end-to-end data validation systems in Hadoop setting for lineage, schema evolution
- nLead system design for Hadoop/Hive test environments, including YARN resource management, dynamic partitioningn




- .Exposure to Zephyr-Jira-ServiceNow integrated test management systems with experience on API-driven automationn
- .Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deploymentsn
- .Create data quality system designs using PySpark integrated with Hive metadata servicesn
- .Design testing platforms, test data generatorn
- sMentor juniors on PySpark testing basics, contribute to testing strategy discussionn
- sSpark session configurations for memory and core allocations for both local and cluster manager settingn
- sData handling with distributed file systems like HDFS and writing back to hive tablen
- sImplementation of Partitioning, caching techniques in organizing code for transformation pipelinen
- sPerformance tuning implementation like salting, minimizing shufflinn

n

g

📌 Pyspark developer (Tamil Nadu)
🏢 Tata Consultancy Services
📍 Tamil Nadu