We are looking for an experienced Python Spark Lead responsible for designing, developing, and optimizing large-scale data processing solutions using Python, PySpark, Apache Spark, SQL, Airflow, and Control-M. The candidate will work on data engineering projects, build scalable ETL pipelines, optimize Spark jobs, and support enterprise data platforms.
Key Responsibilities:
- Develop and maintain scalable data processing pipelines using Python and PySpark.
- Design, build, and optimize Apache Spark applications for large-scale data processing.
- Write complex SQL queries for data extraction, transformation, and analysis.
- Develop and manage ETL workflows using Spark and Python.
- Schedule and monitor workflows using Apache Airflow and Control-M.
- Perform Spark performance tuning, debugging, and optimization.
- Work with large datasets and distributed computing environments.
- Troubleshoot production issues and ensure data pipeline reliability.
- Collaborate with data architects, analysts, and engineering teams.
- Follow best practices for coding, testing, and deployment.