Position: Python Spark Lead

Experience Required:

- 810 Years Total Experience
- Minimum 7 Years Relevant Experience in Python/PySpark/Spark Development

Location:

- Bangalore
- Chennai

Budget: 27 LPA

Role Overview

We are looking for an experienced Python Spark Lead responsible for designing, developing, and optimizing large-scale data processing solutions using Python, PySpark, Apache Spark, SQL, Airflow, and Control-M. The candidate will work on data engineering projects, build scalable ETL pipelines, optimize Spark jobs, and support enterprise data platforms.

Key Responsibilities

- Develop and maintain scalable data processing pipelines using Python and PySpark.
- Design, build, and optimize Apache Spark applications for large-scale data processing.
- Write complex SQL queries for data extraction, transformation, and analysis.
- Develop and manage ETL workflows using Spark and Python.
- Schedule and monitor workflows using Apache Airflow and Control-M.
- Perform Spark performance tuning, debugging, and optimization.
- Work with large datasets and distributed computing environments.
- Troubleshoot production issues and ensure data pipeline reliability.
- Collaborate with data architects, analysts, and engineering teams.
- Follow best practices for coding, testing, and deployment.

Mandatory Skills

- Python
- PySpark
- Apache Spark
- SQL
- Airflow
- Control-M
- ETL/Data Pipeline Development
- Spark Performance Optimization

Job Description: Python Spark Lead

Position: Python Spark Lead





Experience Required:

- 11–15 Years Total Experience
- Minimum 10 Years Relevant Experience in Python/PySpark/Spark Development

Location:

- Bangalore
- Chennai

Budget: 40 LPA

Role Overview

Tech Mahindra is seeking a senior Python Spark Lead with robust expertise in Python, PySpark, Apache Spark, SQL, Airflow, and Control-M to lead enterprise-scale data engineering solutions. The role requires deep experience in Spark architecture, pipeline development, optimization, production support, and technical leadership.

Key Responsibilities

- Lead the design and development of high-performance PySpark and Spark-based data solutions.
- Architect scalable data pipelines using Python, Spark, and SQL.
- Provide technical leadership for complex ETL and data engineering initiatives.
- Optimize Spark jobs for performance, scalability, and reliability.
- Develop and manage enterprise scheduling workflows using Airflow and Control-M.
- Review code, establish development standards, and mentor team members.
- Analyze and resolve complex production issues.
- Implement best practices for data processing, monitoring, and deployment.
- Collaborate with business, architecture, and engineering teams.
- Drive improvements in data platform performance and operational efficiency.

Mandatory Skills

- Python
- PySpark
- Apache Spark
- SQL
- Airflow
- Control-M
- Advanced ETL Development
- Spark Optimization & Performance Tuning
- Data Engineering Leadership

📌 Python spark lead (Chennai)
🏢 Intellics Global Services
📍 Chennai