Roles and Responsibilities

- Architect and scale enterprise lakehouse solutions using Azure Databricks, ADLS Gen2, Delta Lake, and medallion architecture (Bronze, Silver, Gold).
- Design, build, and maintain production-grade batch and streaming ELT/ETL pipelines using PySpark, Spark SQL, Azure Data Factory, and Databricks Workflows.
- Develop configuration-driven, metadata-based ingestion frameworks to rapidly onboard structured, semi-structured, and unstructured data sources.
- Curate and optimize gold-layer data products, vector search indexes, and embeddings via Azure AI Search to support Retrieval-Augmented Generation (RAG) and AI agents.
- Enforce end-to-end data governance, access management, and data lineage utilizing Databricks Unity Catalog and Microsoft Purview to ensure strict compliance (Section 7216, PCAOB, NIST AI RMF).
- Implement performance tuning, Liquid Clustering, Z-Ordering, file compaction, and right-sized compute strategies to minimize platform compute costs.
- Establish CI/CD pipelines, automated data quality testing, data contracts, and proactive observability (freshness, volume,



schema drift).
- Collaborate with AI developers, architects, and cross-functional teams to integrate point solutions into shared, scalable platform services.

Preferred Candidate Profile

Experience

- 4+ years of hands-on experience building production data pipelines and data platforms.
- Proven track record designing lakehouse architectures on Azure cloud infrastructure.

Technical Expertise

- Core Cloud/Data: Azure Databricks, ADLS Gen2, Azure Data Factory, Synapse.
- Languages & Querying: Solid proficiency in Python, PySpark, Spark SQL, and SQL.
- Storage Formats: Delta Lake, Parquet, Avro, JSON.
- AI Grounding: Vector Search, RAG Pipelines, Azure AI Search, Embeddings, Chunking.
- Governance & DevOps: Databricks Unity Catalog, Microsoft Purview, CI/CD (Azure DevOps/GitHub Actions), Infrastructure-as-Code (Terraform/Bicep).

Mandatory : Candidate must be willing to relocate near to Goregaon, Mumbai Location.

📌 AI Data Engineer (Mumbai)
🏢 SG Analytics
📍 Mumbai