Role Overview

We are seeking a Senior Data Engineer to drive cloud data modernization and build scalable, AI-ready data platforms. This role emphasizes expertise in Databricks, PySpark, Azure Data Factory, Logic Apps, and Airflow, with a robust focus on orchestration, pipeline reliability, and end-to-end data workflow management.

Key Responsibilities
Design, build, and maintain scalable data pipelines using Databricks (PySpark), ADF, Logic Apps, and Airflow
Develop and manage end-to-end orchestration frameworks integrating Airflow DAGs with ADF and Logic Apps
Implement advanced workflow orchestration patterns (event-driven, micro-batch, hybrid scheduling)
Ensure pipeline dependency management, execution reliability, and operational excellence
Build high-performance ETL/ELT pipelines using Databricks and Delta Lake architecture
Implement data observability, monitoring, and alerting mechanisms across workflows
Optimize pipelines and workflows for performance, scalability, and cost efficiency
Integrate pipelines with Azure services such as ADLS Gen2, Blob Storage,



and event triggers
Implement CI/CD for pipelines and workflows using GitHub Actions or equivalent
Ensure data quality, governance, and security compliance
Collaborate with data science teams for ML/GenAI data readiness
Mentor junior engineers and drive best practices for orchestration and pipeline design.

Required Qualifications
7+ years of experience in data engineering
Highly proficient in Databricks (PySpark, Delta Lake), Azure Data Factory (ADF), Azure Logic Apps, and Apache Airflow
Exposure to AI/ML and Llm data pipelines and MLflow
Solid programming skills in Python and advanced SQL
Experience in building orchestrated data platforms with multi-tool integration
Solid understanding of ETL/ELT patterns and orchestration frameworks
Experience in cloud-native data architectures (Azure preferred)
Hands-on experience with monitoring, logging, and pipeline reliability

(ref:hirist.tech)

📌 Antino Senior Data Engineer Databricks/azure Gurugram
🏢 Antino Labs
📍 Gurugram