Job DescriptionDear Professionals
NGreetings from Tata consultancy Services,
NJob Title Pyspark Data Engineer
NExperiernce: 6-10 Years
NLocation: Chennai / Kolkata / Hyderabad / Pune
NMode of Work: Work from Office
NJob description
n
- n
- Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing. N
- Architect end-to-end data validation systems in Hadoop setting for lineage, schema evolution N
- Lead system design for Hadoop/Hive test environments, including YARN resource management, dynamic partitioning. N
- Exposure to Zephyr-Jira-ServiceNow integrated test management systems with experience on API-driven automation. N
- Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deployments. N
- Create data quality system designs using PySpark integrated with Hive metadata services. N
- Design testing platforms, test data generators N
- Mentor juniors on PySpark testing basics, contribute to testing strategy discussions N
- Spark session configurations for memory and core allocations for both local and cluster manager settings N
- Data handling with distributed file systems like HDFS and writing back to hive tables N
- Implementation of Partitioning, caching techniques in organizing code for transformation pipelines N
- Performance tuning implementation like salting, minimizing shuffling N