Data Engineer - Databricks & Agentic AI

EXPERIENCE

LOCATION

ROLE TYPE

3-6 years

Gurgaon / Hybrid

Data Engineering

Role Summary

We are looking for a hands-on Data Engineer with strong Databricks experience to join our product and solution engineering team. The primary focus of this role is to design and build reliable lakehouse architectures, data pipelines, data products, and governed data platforms on Databricks across AWS, Azure, and GCP environments.

In addition to core data engineering, the role offers the opportunity to build agentic AI capabilities within Databricks. This includes developing data-aware agents and multi-agent workflows that use governed enterprise data, models, functions, and tools through Unity Catalog and Databricks AI governance capabilities.

This is a Data Engineering role, not a generalist Applied AI role. We are looking for someone whose foundation is Python, SQL, Spark, data modelling, pipeline engineering, and Databricks platform implementation, with the technical depth and interest to extend those skills into production-grade agentic systems.

Key Responsibilities

- Design, build, and maintain scalable lakehouse architectures and data products using Databricks, Delta Lake, Apache Spark, Python, SQL, and cloud object storage.
- Build reliable ingestion pipelines for APIs, files, SFTP, databases, and third-party data sources using Databricks-native and Python-based tooling.
- Develop reusable transformation frameworks and governed data models using Databricks SQL, PySpark, and dbt, with data quality controls, orchestration, and monitoring across Bronze, Silver, and Gold layers.
- Configure and work effectively within Databricks environments, including workspaces, compute, clusters or serverless compute, jobs, notebooks, repositories, secrets, service principals, and deployment workflows.
- Implement data governance, discoverability, lineage, access controls, and secure sharing using Unity Catalog.
- Build data-aware agents and multi-agent workflows within Databricks that can reason over governed enterprise data and invoke approved models, functions, APIs, and tools.
- Use Databricks model serving and AI gateway capabilities to manage model access, routing, usage controls, observability, and guardrails for agentic applications.
- Operationalise data pipelines and agentic workloads through Git, CI/CD, automated testing, environment promotion, and production monitoring.
- Work with solution architects, application engineers, data scientists, and client teams to move solutions from prototypes into secure and supportable production deployments.
- Optimise Databricks workloads for performance, reliability, and cloud cost efficiency.

What We're Looking For

You do not need to be an expert in every AI framework.



Strong Data Engineering and Databricks fundamentals are the priority; agentic AI experience is an important differentiator.
- Approximately 3-6 years of relevant experience in Data Engineering, Data Platform Engineering, or a closely related role.
- Strong hands-on experience with Databricks in a real development or production environment; familiarity with Databricks setup and administration is essential.
- Strong programming skills in Python, SQL, and PySpark, including performance-aware Spark development and debugging.
- Experience designing and implementing lakehouse architectures using Delta Lake and medallion or equivalent layered patterns.
- Hands-on experience with dbt for modular SQL transformations, testing, documentation, lineage, and deployment within a modern data platform.
- Experience building production-grade data ingestion, transformation, orchestration, validation, and monitoring pipelines.
- Practical experience with Unity Catalog, including catalogs, schemas, tables, volumes, permissions, lineage, and governance patterns.
- Understanding of Databricks Workflows or Jobs, cluster and compute configuration, secrets, service principals, repositories, and environment promotion.
- Experience with Git and CI/CD practices for data engineering workloads, including automated testing and deployment.
- Experience with at least one cloud platform: AWS, Azure, or GCP, including object storage, identity, networking, and managed data services.
- Strong troubleshooting skills and the ability to work across data, platform, security, and application boundaries.

Agentic AI Experience

- Hands-on experience building LLM-powered applications, agents, or tool-calling workflows using Python and a suitable framework or SDK.
- Ability to design multi-agent systems with transparent orchestration, state, tool permissions, failure handling, human approval points, and observability.
- Experience grounding agents on governed enterprise data through SQL, vector search, retrieval, functions, APIs, or other controlled tools.
- Understanding of model serving, prompt and response logging, evaluation, tracing, token and cost controls, rate limits, and safety guardrails.
- Familiarity with registering and governing models, agents, functions, connections, and tools through Unity Catalog and Databricks AI governance capabilities.
- Ability to distinguish a production agent from a demo by addressing identity,



permissions, data access, testing, monitoring, resilience, and operational support.

Nice to Have

- Databricks Data Engineer Associate, Data Engineer Professional, or equivalent practical experience.
- Experience with Databricks Asset Bundles, Databricks Connect, MLflow, Model Serving, Vector Search, Lakehouse Monitoring, or Delta Live Tables / Lakeflow Declarative Pipelines.
- Experience with Mosaic AI Agent Framework, LangGraph, LangChain, Semantic Kernel, or another orchestration framework used for production agent systems.
- Experience integrating dbt with Databricks workflows, Unity Catalog, Git-based development, and CI/CD; exposure to Snowflake, Microsoft Fabric, or Airflow is also useful.
- Experience building SaaS or multi-tenant applications, data APIs, or governed client-facing data products.
- Exposure to Docker, serverless containers, cloud networking, private connectivity, firewalls, and enterprise security requirements.

What Success Looks Like

- You can independently build and troubleshoot a Databricks data pipeline from ingestion through curated, governed consumption layers.
- You understand how a Databricks workspace should be organised and secured, not only how to write code in a notebook.
- You can build an agent that uses governed data and tools, and you can explain how its identity, permissions, model access, logging, evaluation, and production support will work.
- You create reusable engineering patterns that help multiple product teams deploy data and agentic solutions faster and more safely.

What We Offer

- Opportunity to work on data, analytics, AI, and digital products across Retail & CPG, Media & Entertainment, Sports, Energy, Telecommunications, and other industries.
- Hands-on work across the Databricks platform, from lakehouse engineering and governance through production-grade agentic AI solutions.
- Exposure to AWS, Azure, GCP, Databricks, Snowflake, and modern cloud-native technologies.
- Opportunity to help establish reusable Databricks and agentic engineering practices for a growing product and solution organisation.
- A development path toward Senior Data Engineer, Databricks Platform Engineer, Data Architect, or technical leadership roles.

About EXL

EXL (NASDAQ:EXLS) is a global data and AI company that helps clients improve business outcomes through data, analytics, AI, digital solutions, and operations. Our teams work with organisations across industries to design, build, and deploy technology-driven solutions addressing complex business problems. Headquartered in New York, EXL has more than 40,000 professionals in locations throughout the United States, Europe, Asia (primarily India and Philippines), Latin America, Australia and South Africa.

📌 Data Engineer - Databricks & Agentic AI (Gurugram)
🏢 EXL
📍 Gurugram