Role- PySpark Developer

Year of Experience- 4 to 15 years

Location -Pune, Hyderabad, Chennai, Bangalore

Technical Skills:
Pyspark
Python concepts and Framework
Spark Architecture
Big Data
SQL

Job Description:

Job Requirements*
Good work experience on Big Data Platforms like Hadoop, PySpark, Scala, Hive, Impala, SQL, Python
Positive Python, Pyspark, Big Data experience
Spark UI/Optimization/debugging techniques
Good python scripting skills
Intermediate SQL exposure – Subquery, Joins, CTE’s
Database technologies

Key Responsibilities
*Design scalable PySpark-based test architectures for ETL/data pipelines, including modular frameworks for batch processing
.Architect end-to-end data validation systems in Hadoop environment for lineage, schema evolutio
nLead system design for Hadoop/Hive test settings, including YARN resource management, agile partitioning




.Exposure to Zephyr-Jira-ServiceNow integrated test management systems with experience on API-driven automation
.Design CI/CD test pipelines for PySpark/Hadoop jobs, incorporating artifact management, parallel execution, and blue-green deployments
.Create data quality system designs using PySpark integrated with Hive metadata services
.Design testing platforms, test data generator
sMentor juniors on PySpark testing basics, contribute to testing strategy discussion
sSpark session configurations for memory and core allocations for both local and cluster manager setting
sData handling with distributed file systems like HDFS and writing back to hive table
sImplementation of Partitioning, caching techniques in organizing code for transformation pipeline
sPerformance tuning implementation like salting, minimizing shufflin

g

📌 Pyspark Developer Chennai (India)
🏢 Tata Consultancy Services
📍 India