We are looking for an experienced Data Engineer with strong hands-on expertise in Apache Airflow to design, develop, and maintain reliable data pipelines and workflows. The ideal candidate should have solid experience in DAG development, workflow orchestration, and production support.
Key Responsibilities
- Design, develop, and maintain Apache Airflow DAGs for data pipeline orchestration.
- Build and manage scalable and reliable ETL/ELT workflows.
- Monitor, troubleshoot, and optimize Airflow DAGs and pipeline performance.
- Implement scheduling, dependencies, retries, alerts, and error-handling mechanisms.
- Work with data engineering teams to integrate Airflow with various data sources and processing platforms.
- Follow best practices for DAG development, testing, deployment, and maintenance.
- Troubleshoot production issues and ensure timely resolution of pipeline failures.
- Maintain proper documentation for workflows, DAGs, and operational processes.
Required Skills
- 4+ years of experience in Data Engineering.
- Strong hands-on experience with Apache Airflow.
- Positive knowledge of DAG development and workflow orchestration.
- Experience developing and managing production-grade data pipelines.
- Understanding of ETL/ELT concepts and data integration.
- Strong troubleshooting and problem-solving skills.
- Knowledge of Python and SQL is preferred.
Certifications Tier 1 – Required for DAG Authors:
- Astronomer Certification for Apache Airflow Fundamentals