Greetings from Tata Consultancy Services (TCS)

We are looking for experienced professionals with strong expertise in PySpark, Big Data Testing, Hadoop Ecosystem, and Data Validation to join our data engineering team.

Experience: 5 to 10 Years

Location: Chennai, Kolkata, Hyderabad, and Pune

Technical Skills Required
Solid experience in Big Data technologies including Hadoop, PySpark, Scala, Hive, Impala, SQL, and Python.
Hands-on expertise in Python scripting and PySpark application development/testing.
Experience in Spark UI analysis, debugging, optimization, and performance tuning.
Good understanding of SQL concepts including Joins, Subqueries, CTEs, and complex query development.
Exposure to database technologies and large-scale data processing environments.
Experience working with distributed systems such as HDFS, Hive, and Hadoop clusters.
Understanding of ETL/Data Pipeline testing methodologies and data quality validation.

Roles & Responsibilities
Design and develop scalable PySpark-based testing frameworks for ETL and data pipeline validation.




Architect end-to-end data validation solutions within Hadoop ecosystems, covering schema validation, lineage checks, and data quality monitoring.
Design and manage Hadoop/Hive test environments, including YARN resource management and dynamic partitioning.
Implement automated testing solutions integrated with Zephyr, Jira, ServiceNow, and API-driven test management tools.
Build and maintain CI/CD test pipelines for PySpark and Hadoop applications with artifact management and parallel execution capabilities.
Develop data quality validation frameworks using PySpark integrated with Hive metadata services.
Create test data generation utilities and reusable testing platforms for large-scale data validation.
Configure and optimize Spark sessions, including memory management and executor/core allocation strategies.
Work extensively with HDFS and Hive tables for distributed data storage and processing.
Implement best practi

📌 Pyspark Developer (Chennai)
🏢 Tata Consultancy Services
📍 Chennai