Role- Py Spark Developer
Year of Experience- 4 to 15 years
Location -Pune, Hyderabad, Chennai, Bangalore
Technical Skills:
Pyspark Python concepts and Framework Spark Architecture Big Data SQL Job Description:
Job Requirements*
Good work experience on Big Data Platforms like Hadoop, Py Spark, Scala, Hive, Impala, SQL, Python Good Python, Pyspark, Big Data experience Spark UI/Optimization/debugging techniques Good python scripting skills Intermediate SQL exposure – Subquery, Joins, CTE's Database technologies Key Responsibilities
*Design scalable Py Spark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing . Architect end-to-end data validation systems in Hadoop environment for lineage, schema evolutio n Lead system design for Hadoop/Hive test environments, including YARN resource management, energetic partitioning .



Exposure to Zephyr-Jira-Service Now integrated test management systems with experience on API-driven automation . Design CI/CD test pipelines for Py Spark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deployments . Create data quality system designs using Py Spark integrated with Hive metadata services . Design testing platforms, test data generator s Mentor juniors on Py Spark testing basics, contribute to testing strategy discussion s Spark session configurations for memory and core allocations for both local and cluster manager setting s Data handling with distributed file systems like HDFS and writing back to hive table s Implementation of Partitioning, caching techniques in organizing code for transformation pipeline s Performance tuning implementation like salting, minimizing shufflin g

📌 Pyspark developer (Chennai)
🏢 Tata Consultancy Services
📍 Chennai