- Design and develop scalable data pipelines using PySpark and Python.
- Develop and optimize large-scale data processing workloads using Apache Spark.
- Work extensively with AWS cloud services for data engineering solutions.
- Build and maintain ETL/ELT pipelines for structured and semi-structured data.
- Perform data transformation, cleansing, aggregation, and validation.
- Optimize PySpark jobs for performance, scalability, and reliability.
- Write complex and optimized SQL queries for data processing and analysis.
- Troubleshoot production issues and improve pipeline performance.
- Collaborate with data engineers, architects, analysts, and other stakeholders.
- Follow coding standards and data-engineering best practices.
Preferred candidate profile
- 612 years of overall IT experience.
- Strong hands-on experience in PySpark-based data engineering.
- Robust practical knowledge of AWS is mandatory.
- Good problem-solving and debugging skills.
- Good communication and stakeholder collaboration skills.