About the role
We are hiring a Senior Azure Databricks Data Engineer to design, build and own an enterprise lakehouse on Azure Databricks — from the first workspace to production-grade Bronze, Silver and Gold layers. You have built a Medallion architecture from the ground up, not just worked inside one. You know how to set up the platform so it scales, performs well and stays cost-efficient, with governance, security and data quality built in from day one.

Key responsibilities
Medallion architecture (Bronze / Silver / Gold)
• Design and build the Medallion lakehouse from scratch on Azure Databricks and Delta Lake.
• Define layer contracts: raw, append-only ingestion in Bronze; cleansed, conformed and deduplicated data in Silver; business-ready, modelled data (star schemas, data marts, aggregates) in Gold.
• Build batch and streaming ingestion with Auto Loader, Structured Streaming, Delta Live Tables (Lakeflow Declarative Pipelines) and Azure Data Factory.
• Implement incremental processing, CDC, SCD Type 1/2, MERGE patterns, schema evolution and late-arriving data handling.
• Set data quality rules and expectations at each layer, with quarantine and alerting for bad records. Platform architecture and scalability
• Architect the Azure Databricks workplace: workspaces, dev/test/prod separation, VNet injection, Private Link, ADLS Gen2 storage layout and naming standards.
• Design for scale across data volume, number of pipelines, teams and concurrent users.
• Define cluster policies, compute types (job, all-purpose, serverless, SQL warehouses) and orchestration with Databricks Workflows.
• Build reusable frameworks: metadata-driven ingestion, common libraries, templates and logging. Performance optimisation
• Tune Spark and Delta workloads: partitioning, liquid clustering, Z-ORDER, OPTIMIZE/VACUUM, file sizing, caching,



AQE and Photon.
• Diagnose and fix skew, shuffle, spill and small-file problems using the Spark UI and query profiles.
• Set and meet SLAs for pipeline run time and data freshness. Cost optimisation (FinOps)
• Right-size clusters, use autoscaling, auto-termination, spot instances and serverless where it lowers cost.
• Enforce cost controls through cluster policies, tagging and budgets.
• Track DBU and Azure spend with system tables and dashboards, and report savings. Governance, security and standards
• Implement Unity Catalog: metastore, catalogs/schemas per layer and domain, external locations, storage credentials.
• Set up role-based access, row
- and column-level security, data masking and PII handling.
• Enable lineage, auditing, data classification and integration with Microsoft Purview.
• Manage secrets with Azure Key Vault and identities with Entra ID service principals and managed identities.
• Define and document engineering standards: coding guidelines, naming conventions, review process and data contracts. DevOps and delivery
• Build CI/CD with Azure DevOps or GitHub Actions and Databricks Asset Bundles.
• Provision infrastructure as code with Terraform or Bicep.
• Add unit and integration tests, monitoring and alerting for pipelines. Leadership
• Act as technical lead: mentor engineers, review code and design, and set best practices.
• Work with architects, analysts,



data scientists and business stakeholders to turn requirements into data products. Required skills and experience
• 7+ years in data engineering, including 3+ years building production workloads on Azure Databricks.
• Proven track record of building a Medallion architecture from the ground up — be ready to walk through one you designed end to end.
• Expert in PySpark and Spark SQL; strong Python and SQL.
• Deep knowledge of Delta Lake: ACID transactions, time travel, MERGE, CDF, OPTIMIZE, liquid clustering.
• Hands-on with Unity Catalog and data governance: access control, lineage, auditing, PII protection.
• Strong Azure stack: ADLS Gen2, Azure Data Factory, Key Vault, Entra ID, Event Hubs, Azure Monitor, networking basics (VNet, Private Endpoints).
• Proven results in Spark performance tuning and cloud cost optimisation, with numbers to show for it.
• Data modelling: dimensional modelling (Kimball), Data Vault awareness, SCD patterns.
• CI/CD and IaC: Git, Azure DevOps or GitHub Actions, Databricks Asset Bundles, Terraform.
• Clear communicator who can explain design trade-offs to both engineers and business stakeholders. Nice to have
• Certifications: Databricks Certified Data Engineer Skilled, Microsoft Certified: Azure Data Engineer Associate (DP-203) or Fabric Data Engineer (DP-700), Databricks Lakehouse Platform Architect.
• Experience migrating from on-prem (SQL Server, Hadoop, Informatica, SSIS) or Synapse to Databricks.
• Databricks SQL, Lakehouse Monitoring, MLflow or Feature Store exposure.
• dbt on Databricks, Great Expectations, or similar data quality tools.
• Power BI integration and semantic model design on Gold data.
• Experience with data mesh or domain-oriented data products. •

📌 Data Analyst, data engineers, data specialist (Pune)
🏢 Diacto Technologies
📍 Pune