Roles and Responsibilities :
- Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure Databricks.
- Collaborate with cross-functional teams to gather requirements for data processing needs and design solutions that meet business needs.
- Develop complex SQL queries on Azure Synapse Analytics to perform real-time analytics tasks such as aggregations, joins, and filtering.
- Ensure high-quality delivery of projects by monitoring project progress, identifying issues early on, and implementing corrective actions.
- Lead a small Data Engineering team (and coordinate partner resources where applicable): planning, tasking, mentoring, code reviews, and delivery governance.
- Design, build, and run scalable ETL/ELT pipelines on Azure Databricks (Spark/Delta) for batch and near real-time processing.
- Deliver curated AI-ready datasets with data contracts: quality checks, refresh cadence, SLAs, and documentation.
- Implement governance & security by design: RBAC, secrets (Key Vault),



auditability/lineage, catalog/metadata, and Dev/UAT/Prod controls (Unity Catalog or equivalent).
- Set up orchestration and automation using ADF and/or Databricks Workflows with retries, backfills, and failure handling.
- Establish CI/CD and engineering standards using Azure DevOps/GitHub Actions: version control, testing, release gates, and rollback.
- Build operability: monitoring, alerting, runbooks, and incident response for critical pipelines.
- Enable AI/GenAI consumption: training/inference feeds, feature-ready data, and document/metadata pipelines for RAG/embeddings refresh.
- Partner with AI Architects, MLOps, Cloud, Security/IRM, and Business SPOCs to manage dependencies and ensure platform readiness.

Job Requirements :
- 4–12 years in Data Engineering with 3+ years leading delivery and owning production pipelines.
- Solid hands-on skills in Python, SQL, PySpark/Spark and data modeling

📌 AI Lead Data Engineer (Navi Mumbai)
🏢 Jio bp
📍 Navi Mumbai