Role & responsibilities
Design, develop, and maintain scalable data pipelines using Python, PySpark, and AWS Glue, EMR, Step function.
Build and optimize distributed data processing workflows using EMR and cluster programming techniques.
Develop Airflow DAGs to orchestrate complex ETL pipelines.
Work extensively with AWS services to build end-to-end data engineering solutions.
Ensure reliability, performance, and scalability of all data workflows.
Troubleshoot production issues quickly and implement corrective actions.
Collaborate with cross-functional teams to deliver high-quality data products.
Independently drive deliverables with minimal supervision.