Role Overview:

Seeking a highly skilled Senior Data Engineer with deep expertise in Databricks, Real-Time Data Processing, Lakehouse Architecture, and AI-driven solutions. The role involves designing, developing, and supporting scalable data platforms and streaming pipelines, leveraging up-to-date Databricks capabilities such as Lakeflow, Unity Catalog, and Generative AI technologies.

Key Responsibilities:

- Design, develop, and maintain end-to-end streaming and batch data pipelines using Databricks and PySpark.
- Build, optimize, and manage Delta Lake tables within Unity Catalog.
- Develop and support Lakeflow pipelines with incremental processing, CDC, and data quality validations.
- Integrate enterprise systems into the Databricks Lakehouse.
- Ensure data quality, governance, security, and operational excellence across platforms.
- Monitor, troubleshoot, and optimize workflows for reliability and performance.
- Collaborate with stakeholders to deliver scalable,



business-aligned solutions.
- Build and support AI-powered data solutions leveraging LLMs, RAG, and vector search.
- Implement CI/CD best practices and maintain deployment pipelines.
- Drive deliverables independently, proactively resolving challenges with minimal supervision.

Tech Stack &

Requirements :

- Hands-on experience with Databricks and modern Lakehouse architecture.
- Expertise in PySpark, Spark Structured Streaming, SQL, NoSQL databases, Delta Lake.
- Proficiency with Unity Catalog for governance and security.
- Experience building AI agents, copilots, and generative AI solutions using LLMs, RAG, and vector search.
- Experience with Git-based source control, CI/CD pipelines, Databricks Repos, and Databricks Asset Bundles.

Educational Qualifications :

- BE/BTech/ME/MTech/MCA only.

📌 Senior Data Engineer - Databricks & AI Solutions (India)
🏢 Hexaconcepts
📍 India