Senior Data Engineer – Databricks & AI Solutions

Experience: 6 Years

Location: Hyderabad / Chennai / Bangalore

Work Mode: Hybrid – 3 Days Work From Office

Shift: Day Shift

Notice Period: 0–30 Days / Serving Notice Period

Role Overview

We are looking for an experienced Senior Data Engineer with strong hands-on expertise in Databricks, Lakehouse Architecture, PySpark, Real-Time Data Processing, AI/GenAI solutions, and automated deployment processes.

The ideal candidate will design, develop, and support scalable batch and streaming data platforms while leveraging modern Databricks capabilities such as Lakeflow, Unity Catalog, Delta Lake, and AI/GenAI technologies.

The candidate should be able to work independently, communicate effectively with business and technical stakeholders, and deliver high-quality solutions with minimal supervision.

Mandatory Skills

Candidates must have strong hands-on experience in:

- Databricks and modern Lakehouse Architecture

- Integration of enterprise data platforms with AI-driven applications

- Automated deployment and release processes / CI/CD

Technical SkillsDatabricks & Data Engineering
- Strong hands-on experience with Databricks

- Modern Lakehouse Architecture

- PySpark

- Spark Structured Streaming

- SQL

- NoSQL databases

- Delta Lake

- Unity Catalog

- Databricks Workflows and Job Orchestration

- Bronze, Silver, and Gold layer architecture

- Batch and real-time data processing

Databricks Lakeflow

Hands-on experience with:

- Lakeflow

- Declarative Pipelines

- Lakeflow Connect

- Change Data Capture (CDC)

- Incremental Data Processing

- Data Quality Validation





- Pipeline monitoring and optimization

AI / Generative AI

Experience building or integrating

- AI Agents

- Copilots

- Generative AI solutions

- Large Language Models (LLMs)

- Retrieval-Augmented Generation (RAG)

- Vector Search

- Prompt Engineering

Candidates should understand how enterprise data platforms can be integrated with AI-powered applications.

Data Governance & Security

Experience with

- Unity Catalog

- Role-Based Access Control (RBAC)

- Data Masking

- Data Protection

- Audit Logging

- Data Lineage

- Data Governance

- Compliance

DevOps & Deployment

Hands-on experience with:

- Git-based source control

- CI/CD

- Databricks Repos

- Databricks Asset Bundles (DABs)

- Automated deployment and release processes

Key Responsibilities
- Design, develop, and maintain end-to-end batch and streaming data pipelines using Databricks and PySpark.

- Build and optimize Delta Lake tables within Unity Catalog.

- Develop and support Lakeflow pipelines using CDC, incremental processing, and data quality validations.

- Integrate enterprise systems such as SharePoint and other external data sources into the Databricks Lakehouse.

- Implement data governance, security, data quality, lineage,



and compliance standards.

- Monitor, troubleshoot, and optimize data workflows for reliability and performance.

- Work with technical and business stakeholders to understand requirements and deliver scalable data solutions.

- Build and support AI-powered data solutions, copilots, and intelligent agents using LLMs, RAG, and vector search.

- Implement CI/CD best practices and automated deployments using Databricks Repos and Asset Bundles.

- Independently drive deliverables and proactively identify and resolve technical challenges.

Preferred Qualifications
- Experience working on large-scale enterprise data platforms.

- Strong understanding of Lakehouse and Data Mesh concepts.

- Exposure to Azure cloud services.

- Experience with enterprise integration patterns.

- Robust problem-solving and analytical skills.

- Excellent stakeholder management and communication skills.

Candidate Requirements
- Minimum 6 years of relevant experience.

- Candidate must currently be located in Hyderabad, Chennai, or Bangalore.

- Must be comfortable working 3 days from office.

- Must be willing to move to onsite work in the future if required.

- Notice period must be 0–30 days or candidate should already be serving notice.

- Strong employment stability is required. Candidates with frequent job changes should not apply.

- Prefer candidates with at least 2 years of stability in an organization.

- Excellent communication skills are mandatory.

- Candidates should be available for interview coordination within 24–72 hours.

📌 Senior Data Engineer – Databricks & AI Solutions (Bengaluru)
🏢 Umanist Staffing
📍 Bengaluru