Dear ProfessionalsGreetings from Tata consultancy Services,Job Title Pyspark Data EngineerExperiernce: 6-10 YearsLocation: Chennai / Kolkata / Hyderabad / PuneMode of Work : Work from Office

Job description

Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing.Architect end-to-end data validation systems in Hadoop environment for lineage, schema evolutionLead system design for Hadoop/Hive test environments, including YARN resource management, agile partitioning.Exposure to Zephyr-Jira-ServiceNow integrated test management systems with experience on API-driven automation.Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution,



and blue-green deployments.Create data quality system designs using PySpark integrated with Hive metadata services.Design testing platforms, test data generatorsMentor juniors on PySpark testing basics, contribute to testing strategy discussionsSpark session configurations for memory and core allocations for both local and cluster manager settingsData handling with distributed file systems like HDFS and writing back to hive tablesImplementation of Partitioning, caching techniques in organizing code for transformation pipelinesPerformance tuning implementation like salting, minimizing shuffling

📌 Pyspark Data Engineer (Mumbai)
🏢 TCS
📍 Mumbai