What are we looking for We are seeking an experienced and highly skilled Airflow Orchestration Specialist to play a pivotal role in our enterprise data modernization programs. This role will focus on migrating legacy workflows from Oozie to Airflow, orchestrating data pipelines on AWS, and optimizing data ingestion and transformation processes leveraging Databricks.
Key Responsibilities:
Workflow Migration:
- Analyze and convert Oozie workflows into Airflow DAGs using Python-based orchestration.
- Design and implement reusable, modular, and optimized Airflow pipelines for data ingestion, transformation, and orchestration.
- Maintain a one-to-one mapping between legacy workflows and Airflow DAGs, ensuring no data loss or business interruption.

Cloud Data Migration:
- Collaborate with data engineering teams to migrate Cloudera Hadoop workloads to Databricks on AWS.
- Leverage Airflow for scheduling and orchestrating data workflows on AWS-based services (e.g., S3, EMR, Glue, Redshift).

Pipeline Optimization:




- Optimize data ingestion pipelines to achieve high throughput and low latency on AWS cloud infrastructure.
- Integrate Airflow workflows with Databricks for data transformations and analytics.

Error Handling and Monitoring:
- Implement robust error-handling mechanisms, task retries, and alerting within Airflow workflows.
- Set up monitoring dashboards using tools like CloudWatch, Prometheus, or Airflow's built-in features.

Collaboration and Documentation:
- Work closely with data architects, cloud engineers, and DevOps teams to align on migration goals and architecture.
- Document the migration process, workflow logic, and best practices for ongoing maintenance.

Performance Testing:
- Conduct performance testing and benchmarking of migrated pipelines to ensure productive resource utilization on Databricks and AWS.

CI/CD Implementation:
- Design and maintain CI/CD pipelines for orchestrated workflows using Jenkins and Terraform.
- Automate deployment of Airflo

📌 Airflow Orchestration (India)
🏢 Brighttier
📍 India