Senior Data Engineer – Databricks & AI Solutions

Experience: 6+ Years
Location: Hyderabad / Chennai / Bangalore
Work Mode: Hybrid – 3 Days Work From Office
Shift: Day Shift
Notice Period: 0–30 Days / Serving Notice Period

Role Overview

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Databricks, Lakehouse Architecture, PySpark, Real-Time Data Processing, AI/GenAI solutions, and automated deployment processes.

The ideal candidate will design, develop, and support scalable batch and streaming data platforms while leveraging up-to-date Databricks capabilities such as Lakeflow, Unity Catalog, Delta Lake, and AI/GenAI technologies.

The candidate should be able to work independently, communicate effectively with business and technical stakeholders, and deliver high-quality solutions with minimal supervision.

Mandatory Skills

Candidates must have strong hands-on experience in:

-

Databricks and modern Lakehouse Architecture

-

Integration of enterprise data platforms with AI-driven applications

-

Automated deployment and release processes / CI/CD

Technical SkillsDatabricks & Data Engineering

-

Strong hands-on experience with Databricks

-

Modern Lakehouse Architecture

-

PySpark

-

Spark Structured Streaming

-

SQL

-

NoSQL databases

-

Delta Lake

-

Unity Catalog

-

Databricks Workflows and Job Orchestration

-

Bronze, Silver, and Gold layer architecture

-

Batch and real-time data processing

Databricks Lakeflow

Hands-on experience with:

-

Lakeflow

-

Declarative Pipelines

-

Lakeflow Connect

-

Change Data Capture (CDC)

-

Incremental Data Processing

-

Data Quality Validation

-





Pipeline monitoring and optimization

AI / Generative AI

Experience building or integrating:

-

AI Agents

-

Copilots

-

Generative AI solutions

-

Large Language Models (LLMs)

-

Retrieval-Augmented Generation (RAG)

-

Vector Search

-

Prompt Engineering

Candidates should understand how enterprise data platforms can be integrated with AI-powered applications.

Data Governance & Security

Experience with:

-

Unity Catalog

-

Role-Based Access Control (RBAC)

-

Data Masking

-

Data Protection

-

Audit Logging

-

Data Lineage

-

Data Governance

-

Compliance

DevOps & Deployment

Hands-on experience with:

-

Git-based source control

-

CI/CD

-

Databricks Repos

-

Databricks Asset Bundles (DABs)

-

Automated deployment and release processes

Key Responsibilities

-

Design, develop, and maintain end-to-end batch and streaming data pipelines using Databricks and PySpark.

-

Build and optimize Delta Lake tables within Unity Catalog.

-

Develop and support Lakeflow pipelines using CDC, incremental processing, and data quality validations.

-

Integrate enterprise systems such as SharePoint and other external data sources into the Databricks Lakehouse.

-

Implement data governance, security, data quality, lineage,



and compliance standards.

-

Monitor, troubleshoot, and optimize data workflows for reliability and performance.

-

Work with technical and business stakeholders to understand requirements and deliver scalable data solutions.

-

Build and support AI-powered data solutions, copilots, and intelligent agents using LLMs, RAG, and vector search.

-

Implement CI/CD best practices and automated deployments using Databricks Repos and Asset Bundles.

-

Independently drive deliverables and proactively identify and resolve technical challenges.

Preferred Qualifications

-

Experience working on large-scale enterprise data platforms.

-

Strong understanding of Lakehouse and Data Mesh concepts.

-

Exposure to Azure cloud services.

-

Experience with enterprise integration patterns.

-

Strong problem-solving and analytical skills.

-

Excellent stakeholder management and communication skills.

Candidate Requirements

-

Minimum 6+ years of relevant experience.

-

Candidate must currently be located in Hyderabad, Chennai, or Bangalore.

-

Must be comfortable working 3 days from office.

-

Must be willing to move to onsite work in the future if required.

-

Notice period must be 0–30 days or candidate should already be serving notice.

-

Strong employment stability is required. Candidates with frequent job changes should not apply.

-

Prefer candidates with at least 2 years of stability in an organization.

-

Excellent communication skills are mandatory.

-

Candidates should be available for interview coordination within 24–72 hours.

📌 Senior Data Engineer Databricks & AI Solutions (Bengaluru)
🏢 Important Business
📍 Bengaluru