We are looking for a skilled PySpark Developer with up to 3 years of experience in building and maintaining large-scale data processing pipelines. The ideal candidate should have hands-on experience in PySpark, Python, SQL, and Big Data technologies and be capable of developing efficient ETL workflows for enterprise data platforms.
Responsibilities
- Design, develop, and maintain data pipelines using PySpark.
- Develop ETL/ELT processes for ingesting, transforming, and loading large volumes of data.
- Write optimized PySpark code for data processing and transformation.
- Work with structured and semi-structured data from multiple sources.
- Develop and optimize SQL queries for data extraction and validation.
- Troubleshoot data quality and performance issues.
- Collaborate with Data Engineers, Analysts, and Business teams to understand requirements.
- Participate in code reviews and follow data engineering best practices.
- Monitor and support production data pipelines.
Technical Requirements
- Strong experience in PySpark.
- Good programming knowledge of Python.
- Hands-on experience with SQL.
- Understanding of ETL/ELT concepts and data warehousing.
- Experience working with large datasets and distributed processing.
- Knowledge of Spark SQL, DataFrames, and Spark transformations.
- Familiarity with Linux/Unix workplace.
Additional Responsibilities
- Exposure to cloud platforms such as AWS, Azure, or GCP.
- Knowledge of Databricks.
- Experience with workflow orchestration tools such as Airflow.
- Understanding of CI/CD concepts.
Educational Requirements
- Bachelor Of Comp. Applications
- Bachelor Of Computer Science
- Bachelor Of Science
- Bachelor Of Engineering
- Bachelor Of Technology
Preferred Skills
- Technology->Big Data - Data Processing->PySpark