Dear Professionals /n Greetings from Tata consultancy Services, /n Job Title Pyspark Data Engineer /n Experiernce: 6-10 Years /n Location: Chennai / Kolkata / Hyderabad / Pune /n Mode of Work : Work from Office /n /n /n

- Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing.

/n

- Architect end-to-end data validation systems in Hadoop environment for lineage, schema evolution

/n

- Lead system design for Hadoop/Hive test environments, including YARN resource management, agile partitioning.

/n

- Exposure to Zephyr-Jira-ServiceNow integrated test management systems with experience on API-driven automation.

/n

- Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deployments.

/n





- Create data quality system designs using PySpark integrated with Hive metadata services.

/n

- Design testing platforms, test data generators

/n

- Mentor juniors on PySpark testing basics, contribute to testing strategy discussions

/n

- Spark session configurations for memory and core allocations for both local and cluster manager settings

/n

- Data handling with distributed file systems like HDFS and writing back to hive tables

/n

- Implementation of Partitioning, caching techniques in organizing code for transformation pipelines

/n

- Performance tuning implementation like salting, minimizing shuffling

/n

📌 Pyspark Data Engineer (Bengaluru)
🏢 Tata Consultancy Services
📍 Bengaluru