17 Aug
|
PropStream
|
Delhi
Role Overview
We are looking for a hands-on, senior Databricks Architect to design, build, and govern our Lakehouse data platform from the ground up. You will own the end-to-end architecture of our data infrastructure — from raw ingestion through the Medallion layers to serving — and establish the engineering standards that will guide the entire data organization.
This is a highly strategic and technical role focused on driving adoption of Databricks, Unity Catalog, and up-to-date Lakehouse patterns across all data products and pipelines.
Key Responsibilities
Lakehouse Architecture & Design
Design and implement a production-grade Medallion Architecture (Bronze / Silver / Gold) across all data pipelines.
Establish best practices for Delta Lake table design, partitioning strategies, Z-ordering, and optimization across large-scale datasets.
Define data modeling standards and schema evolution policies across the Lakehouse.
Architect end-to-end data flows from ingestion (streaming and batch) through transformation and serving layers.
Unity Catalog & Data Governance
Lead the setup,
configuration, and rollout of Unity Catalog as the centralized governance layer for all data assets.
Design metastore hierarchy, catalog/schema/table organization, and tagging standards.
Implement fine-grained access control (row-level, column-level), data masking policies, and audit logging.
Establish data lineage tracking and ensure end-to-end visibility across all pipelines.
Define and enforce data classification and sensitivity frameworks for PII and regulated data assets.
Pipeline Development & Orchestration
Build and maintain production-grade data pipelines using Py Spark, Delta Live Tables (DLT), and Databricks Workflows / Jobs.
Design modular, reusable pipeline patterns including incremental ingestion, CDC (Change Data Capture), and full-refresh strategies.
Implement robust pipeline observability: logging, alerting, lineage tracking, and SLA monitoring.
Leverage Databricks Repos for CI/CD i
📌 Data engineer (Delhi)
🏢 PropStream
📍 Delhi