Role & responsibilities

Airflow Developer / Data Engineer with deep expertise in Azure Databricks to design, build, and orchestrate our next-generation data pipelines. In this role, you will be responsible for developing scalable ETL/ELT workflows, managing complex DAGs (Directed Acyclic Graphs) in Apache Airflow, and optimizing big data processing workloads within the Azure ecosystem.

Workflow Orchestration & Airflow Management

- DAG Development: Design, write, and maintain complex Apache Airflow DAGs using clean, reusable, and modular Python code.
- Custom Operators: Build and implement custom Airflow operators, hooks, and sensors to integrate with proprietary or niche systems.
- Performance Tuning: Optimize Airflow scheduling, concurrency limits, and task execution to prevent bottlenecks and reduce infrastructure costs.
- Monitoring & Alerting: Implement robust logging, error handling, and alerting mechanisms (e.g., Slack, PagerDuty, email notifications) for pipeline failures.

Data Engineering & Azure Databricks

- Pipeline Development: Design and deploy scalable data pipelines using PySpark / Delta Lake within Azure Databricks.
- Databricks Integration: Use Airflows DatabricksSubmitRunOperator or DatabricksRunNowOperator to orchestrate Databricks Notebooks and multi-task workflows smoothly.
- Performance Optimization: Optimize Spark jobs by configuring cluster sizes, addressing data skew,



caching efficiently, and managing file sizing (e.g., Z-Ordering, Optimize).
- Data Lakehouse Architecture: Maintain Medallion Architecture (Bronze, Silver, Gold layers) inside Azure Data Lake Storage (ADLS Gen2).

Cloud Infrastructure & DevOps

- CI/CD Automation: Build and automate deployment pipelines for Airflow DAGs and Spark code using Azure DevOps or GitHub Actions.
- Access & Security: Configure secure connections between Airflow and Azure services using Azure Key Vault, Managed Identities, and Service Principals.
- Environment Maintenance: Collaborate with platform teams to manage Airflow environments (e.g., Astronomer, MWAA, Docker/Kubernetes containerized setups).

Preferred candidate profile

Technical Skills

- Apache Airflow: 3+ years of hands-on experience orchestrating enterprise-grade data pipelines. Deep knowledge of Airflow 2.x features (TaskFlow API, Energetic Task Mapping).
- Azure Databricks & Spark: 3+ years of experience processing large-scale datasets with PySpark, Spark SQL, and Delta Lake.
- Programming: Advanced proficiency in Python and SQL.
- Azure Cloud Platform: Solid familiarity with Azure Data Lake Storage (ADLS Gen2), Azure Key Vault, and Azure DevOps.
- Data Modeling: Strong understanding of data warehousing concepts, star/snowflake schemas, and slow-changing dimensions (SCD).

📌 Airflow Developer (Hyderabad)
🏢 Genpact
📍 Hyderabad