Interested candidates can contact us using the details below.

Contact Number: (phone hidden)

Email Your Updated CV To: [email protected]

Please mention Python Spark Developer in the email subject line when sharing your resume.

Job Title: Python Spark Developer

Experience: 4–14 Years
Work Mode: Hybrid – 12 Days a Month
Employment Type: Full-Time
Key Skills: Python, PySpark, Apache Spark, Hadoop, HDFS, Hive, Apache Airflow, SQL, Unix/Linux, Apache Iceberg

Job Description

We are looking for an experienced Python Spark Developer with strong expertise in big data engineering and the Hadoop ecosystem. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and ETL workflows using Python, PySpark, Spark, Hadoop, and related technologies.

The candidate should have strong SQL skills, experience with workflow orchestration using Airflow, and knowledge of Apache Iceberg.

Roles and Responsibilities

- Design, develop, and maintain scalable ETL and data processing pipelines using Hadoop ecosystem technologies.
- Build and optimize large-scale data processing jobs using Apache Spark, PySpark, and Hive.
- Develop, schedule, and manage workflows using Apache Airflow.
- Write complex SQL and HiveQL queries for data transformation and reporting.




- Integrate structured and unstructured data from multiple sources, including Kafka, Sqoop, APIs, and files.
- Work with HDFS and distributed data processing systems.
- Implement data quality checks, data lineage, and governance standards.
- Monitor, troubleshoot, and optimize Spark jobs for performance and scalability.
- Work with Apache Iceberg for modern data lake and table management requirements.
- Maintain technical documentation and follow coding best practices.
- Collaborate with business and technical teams to deliver reliable data solutions.

Required Skills

- Strong hands-on experience in Python and PySpark.
- Positive knowledge of Apache Spark, Hadoop, HDFS, and Hive.
- Strong SQL skills and data modeling fundamentals.
- Hands-on experience with Apache Airflow.
- Strong knowledge of Apache Iceberg.
- Good understanding of Unix/Linux and shell scripting.
- Knowledge of distributed computing and big data architecture.
- Familiarity with Git, version control, and Agile delivery practices.
- Experience in batch and/or streaming data processing.

Preferred Candidate Profile

- Relevant Experience: 4–14 Years
- Strong analytical, debugging, and problem-solving skills.
- Ability to develop and maintain high-performance data pipelines.
- Good communication and collaboration skills.
-

📌 Python + Spark Developer Hadoop + Pyspark + Python + Airflow + SQL (Chennai)
🏢 Metrixit Solutions
📍 Chennai