Databricks Data Engineer (Unity Catalog, Lakeflow, PySpark)

Location: India - 100% Remote
Engagement: Contract
Experience: 6+ years

About the Role

We are seeking an experienced Databricks Data Engineer to design, build and maintain scalable data pipelines and governed data solutions on the Databricks Lakehouse platform - hands-on with Databricks Jobs and Lakeflow pipelines, Unity Catalog, SQL warehouses and enterprise data security - delivering production-grade pipelines for demand forecasting and retail performance intelligence.

Key Responsibilities

- Design, develop and maintain Databricks Jobs and Lakeflow pipelines: ingestion, silver layer transformation, feature engineering, model training and forward projection
- Implement and manage Unity Catalog - schemas, tables, volumes, lineage and access grants - as the single governance layer
- Build and optimise Databricks SQL warehouses for Genie Spaces, dashboards and ad-hoc performance queries
- Enforce security with Unity Catalog RBAC - column-level, row-level and workspace-level permissions
- Develop the silver layer - ingestion from SQL Server and data quality checks
- Build feature engineering notebooks: weekly demand aggregation, SKU attributes, demand calendar, store traffic and horizon training features
- Work with ML engineers on model pipeline integrity (eval metrics, routing correctness, production refits with recent data)
- Monitor pipeline health,



validate data quality gates and support stakeholder validation milestones
- Maintain notebook/pipeline documentation, Git version control and reproducible run configurations

Required Skills & Qualifications
- Strong hands-on Databricks - Jobs, Lakeflow (Delta Live Tables), notebooks, clusters, SQL warehouses
- PySpark and Databricks SQL - complex transformations, window functions, aggregations, performance tuning
- Deep Unity Catalog experience - three-level namespace, access control, lineage, cross-workspace governance
- Databricks security model - workspace access, cluster policies, secrets, row/column-level security
- ELT pipelines from SQL Server and other relational sources into Delta tables (bronze, silver, gold)
- Delta Lake - ACID, merge patterns, schema evolution, time travel
- Strong debugging and data quality skills
- Git, CI/CD for notebooks and pipelines, Agile delivery
- Excellent communication with technical and business stakeholders

Preferred (Positive to Have)
- Databricks Genie Spaces / natural language query interfaces
- Retail demand forecasting pipelines
- Databricks Certified Data Engineer Associate or Professional
- Power BI with Databricks SQL warehouses
- SQL Server ingestion (pyodbc, JDBC) and incremental loads
- Data quality frameworks - expectations, row counts, freshness checks

Education

Bachelor's degree

📌 Databricks Data Engineer (Unity Catalog, Lakeflow, PySpark) (India)
🏢 Digital Links
📍 India