Job Summary :

We are looking for a Senior Data Engineer with 6 years of experience in Databricks, real-time data processing, Lakehouse architecture, and AI-driven solutions. The role involves designing scalable batch and streaming data platforms and delivering high-quality, governed data and AI solutions.

Key Responsibilities :

- Design, develop, and maintain batch and streaming data pipelines using Databricks and PySpark.
- Build, optimize, and manage Delta Lake tables within Unity Catalog.
- Develop and support Lakeflow pipelines with incremental processing, CDC, and data quality validation.
- Integrate SharePoint and other external enterprise data sources into the Databricks Lakehouse.
- Ensure data quality, governance, security, reliability, and operational excellence.
- Monitor, troubleshoot, and optimize data workflows for performance and reliability.
- Collaborate with business and technical stakeholders to deliver scalable solutions.
- Build AI-powered data solutions, copilots, and intelligent agents using LLMs, RAG, and vector search.
- Implement CI/CD and automated deployment using Databricks Repos and Asset Bundles.
- Drive deliverables independently and proactively resolve technical challenges.

Mandatory Skills :

- 6 years of Data Engineering experience.
- Strong hands-on experience with Databricks and modern Lakehouse architecture.
- Strong proficiency in PySpark,



Spark Structured Streaming, and SQL.
- Experience with Delta Lake, Unity Catalog, Databricks Workflows, and job orchestration.
- Experience with Bronze, Silver, and Gold Lakehouse architecture.
- Practical experience with Databricks Lakeflow, Declarative Pipelines, Lakeflow Connect, CDC, data quality, and incremental processing.
- Experience integrating enterprise data platforms with AI-driven applications.
- Experience with LLMs, RAG, vector search, and prompt engineering.
- Strong understanding of automated deployment and release processes.
- Experience with Git-based source control and CI/CD.

Data Governance & Security :

- Unity Catalog governance and security.
- Role-Based Access Control (RBAC).
- Data masking and data protection policies.
- Audit logging, data lineage, and compliance.

DevOps & Deployment :

- Git-based source control and CI/CD.
- Databricks Repos.
- Databricks Asset Bundles (DABs).
- Automated deployment and release processes.

Preferred Skills :

- Experience with large-scale enterprise data platforms.
- Robust understanding of Lakehouse and Data Mesh principles.
- Exposure to Azure cloud services and enterprise integration patterns.
- Experience with NoSQL databases.
- Strong problem-solving, communication, and stakeholder management skills.

📌 Senior Data Engineer - Databricks & AI Solutions (India)
🏢 Okda Solutions
📍 India