Key Responsibilities:

- Key Responsibilities
- Develop and maintain data pipelines using PySpark
- Process and analyze large scale datasets in distributed environments
- Design and implement ETL ELT workflows
- Optimize Spark jobs for performance and scalability
- Work with data stored in HDFS Hive or cloud storage S3 ADLS
- Collaborate with data engineers analysts and business teams
- Ensure data quality integrity and governance
- Debug and troubleshoot data processing issues
- Automate workflows using scheduling tools Airflow Oozie etc
- Write clean scalable and efficient code
- Required Skills Qualifications
- Technical Skills
- Solid proficiency in Python and PySpark
- Good experience with Apache Spark RDDs DataFrames Spark SQL
- Knowledge of Hadoop ecosystem HDFS Hive
- Experience in ETL pipeline development
- Familiarity with SQL and database concepts
- Experience with data formats Parquet ORC JSON CSV
- Basic understanding of distributed computing concepts
- Exposure to version control tools Git

Technical Requirements:

- Technology Big Data Data Processing PySpark

Preferred Skills:

Technology->Big Data - Data Processing->PySpark

📌 PySpark Developer (India)
🏢 Infosys
📍 India