Role Overview

We are looking for a Senior Databricks & Governance Engineer with 5+ years of Data Engineering experience to develop scalable data pipelines, implement enterprise data governance, and prepare unstructured data for Generative AI applications in a modern Lakehouse setting.

Key Responsibilities

- Develop and maintain batch and streaming ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake.
- Orchestrate data workflows using Azure Data Factory (ADF).
- Extract and process unstructured data from PDFs, images, logs, and text files.
- Prepare data for RAG, LLM applications, embeddings, and vector search.
- Implement data access controls, security, and lineage using Unity Catalog.
- Integrate Databricks metadata with Collibra using REST APIs.
- Design and maintain Bronze, Silver, and Gold layers using Medallion Architecture.
- Develop dimensional and Lakehouse data models.
- Optimize performance using Z-Ordering, Liquid Clustering,



and Delta Lake optimization techniques.
- Ensure data quality, governance, metadata consistency, and pipeline reliability.

Mandatory Skills

- 5+ years of Data Engineering experience
- Strong hands-on experience with Azure Databricks
- PySpark, Python, SQL, and Delta Lake
- Azure Data Factory (ADF)
- Unity Catalog — governance, access controls, and lineage
- Collibra REST API integrations — metadata extraction and synchronization
- GenAI / LLM data preparation, including unstructured data processing
- RAG, embeddings, and vector search
- Lakehouse and Medallion Architecture
- Dimensional data modeling and performance optimization

Skills: pyspark,azure data factory,collibra,genai,delta lake,unity catalog,sql,python,azure databricks

📌 Senior Databricks & Governance Engineer (India)
🏢 Papigen
📍 India