Role

AI Data Platform Engineer - Databricks Experience Guide

- 5-10 years Primary Skill Area
- Genie, AI/BI, Mosaic AI, Unity Catalog & Modern Data Platforms

The chance

Build and operate enterprise-scale Data & AI platforms leveraging Databricks Lakehouse architecture. The role focuses on scalable data engineering, reusable platform engineering, governed self-service analytics, AI-enabled data products, APIs, enterprise service integration, Git-based delivery, Data SRE, data security, Immuta-style governance, and agentic automation using Databricks-native services including Genie, AI/BI, Mosaic AI, Unity Catalog, Vector Search, Lakeflow, and Delta Lake.

Your key responsibilities

Databricks Data Engineering

- Design and implement scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Lake, Databricks Workflows, Lakeflow, and Delta Live Tables.
- Develop reusable ingestion frameworks supporting batch, streaming, event-driven, CDC, file-based, and API-based processing patterns.
- Build curated, analytics-ready, and AI-ready data products with strong quality, lineage, semantic context, and operational controls.
- Optimise workloads for performance, cost, cluster/serverless usage, storage layout, and reliability.

Lakehouse & Platform Engineering

- Create reusable platform accelerators for workspace onboarding, pipeline templates, deployment standards, logging, monitoring, and support runbooks.
- Implement Git connectivity, branching strategy, pull requests, code reviews, CI/CD, deployment bundles, and controlled environment promotion.
- Integrate Databricks with enterprise APIs, source systems, orchestration platforms, governance tools, security services, and downstream analytics consumers.
- Contribute to architecture reviews, technical design documentation, release management,



and platform operations.

Genie, AI/BI & Agentic Enablement

- Configure and manage Databricks Genie Spaces, AI/BI dashboards, and governed natural-language analytics over trusted data products.
- Enable Mosaic AI, MLflow, Vector Search, RAG, GraphRAG, agentic workflows, semantic retrieval, and AI-ready data products.
- Apply AI-assisted operations for schema drift detection, anomaly detection, pipeline failure diagnosis, automated documentation, and data quality recommendations.

Governance, Security & Data SRE

- Implement Unity Catalog governance including RBAC/ABAC, lineage, audit logging, data masking, privacy controls, policy enforcement, and AI governance.
- Integrate with Immuta, Microsoft Purview, IAM, secrets management, monitoring tools, and enterprise access workflows.
- Build Data SRE capabilities covering observability, incident management, restartability, SLA/SLO tracking, root-cause analysis, FinOps, and production readiness.

Skills and attributes for success

Skill / capability area - Details

- Core platform - Databricks Lakehouse, Delta Lake, Lakeflow, Delta Live Tables, Databricks Workflows, Databricks SQL, Unity Catalog.
- AI and GenAI - Databricks Genie, AI/BI, Mosaic AI, MLflow, Vector Search, RAG, GraphRAG, Agentic AI, LLMOps, APIs expertise
- Engineering - Python, PySpark, SQL, APIs, Git, CI/CD, unit testing, integration testing, data pipeline testing,



GitHub Copilot.
- Cloud and DevOps - AWS, Azure or GCP, Terraform/OpenTofu, Kubernetes, Docker, GitHub Actions, Azure DevOps, Jenkins, policy-as-code, Databricks Asset Bundle
- Governance and reliability - Unity Catalog, Immuta, Purview, lineage, audit, masking, data quality, observability, Data SRE, FinOps, Responsible AI, AI security.

To qualify for the role, you must have

- 5-10 years of experience in data engineering, data platform operations, analytics engineering, platform engineering, or AI platform enablement.
- Strong hands-on implementation experience with cloud data platforms, APIs, Git connectivity, CI/CD, governed access patterns, SRE practices, and production operations.
- Preferred certifications aligned to the relevant cloud/platform stack, data engineering, DevOps, security, governance, and AI/ML engineering.

Ideally, you'll also have

- Strong hands-on engineer with architecture awareness, delivery ownership, and a platform engineering mindset.
- Comfortable turning platform standards into reusable frameworks, secure implementation patterns, operational controls, and production-ready services.
- Able to mentor engineers, collaborate with architects/security/SRE teams, and adopt newer AI-native and agentic engineering methods.

What we look for

- Strong hands-on engineer with architecture awareness, delivery ownership, and a platform engineering mindset.
- Comfortable turning platform standards into reusable frameworks, secure implementation patterns, operational controls, and production-ready services.
- Able to mentor engineers, collaborate with architects/security/SRE teams, and adopt newer AI-native and agentic engineering methods.

📌 Data Engineer (Kolkata)
🏢 EY
📍 Kolkata