Job Summary

We are looking for a Senior Databricks Governance Engineer with 5+ years of Data Engineering experience to develop scalable data pipelines, implement enterprise data governance, and prepare unstructured data for Generative AI applications in a contemporary Lakehouse environment.

Responsibilities

- Develop and maintain batch and streaming ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake.
- Orchestrate data workflows using Azure Data Factory (ADF) .
- Extract and process unstructured data from PDFs, images, logs, and text files.
- Prepare data for RAG, LLM applications, embeddings, and vector search .
- Implement data access controls, security, and lineage using Unity Catalog.
- Integrate Databricks metadata with Collibra using REST APIs.
- Design and maintain Bronze, Silver, and Gold layers using Medallion Architecture.
- Develop dimensional and Lakehouse data models.
- Optimize performance using Z-Ordering, Liquid Clustering, and Delta Lake optimization techniques .




- Ensure data quality, governance, metadata consistency, and pipeline reliability.

Mandatory Skills

- 5+ years of Data Engineering experience
- Strong hands-on experience with Azure Databricks
- PySpark, Python, SQL, and Delta Lake
- Azure Data Factory (ADF)
- Unity Catalog - governance, access controls, and lineage
- Collibra REST API integrations - metadata extraction and synchronization
- GenAI / LLM data preparation, including unstructured data processing
- RAG, embeddings, and vector search
- Lakehouse and Medallion Architecture
- Dimensional data modeling and performance optimization

Disclaimer: This job posting and Location has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Sr Databricks & Governance Engineer (India)
🏢 Papigen
📍 India