About the roleWe are hiring a Senior Azure Databricks Data Engineer to design, build and own an enterprise lakehouse on Azure Databricks — from the first workspace to production-grade Bronze, Silver and Gold layers. You have built a Medallion architecture from the ground up, not just worked inside one. You know how to set up the platform so it scales, performs well and stays cost-efficient, with governance, security and data quality built in from day one.

Key responsibilitiesMedallion architecture (Bronze / Silver / Gold)

- Design and build the Medallion lakehouse from scratch on Azure Databricks and Delta Lake.
- Define layer contracts: raw, append-only ingestion in Bronze; cleansed, conformed and deduplicated data in Silver; business-ready, modelled data (star schemas, data marts, aggregates) in Gold.
- Build batch and streaming ingestion with Auto Loader, Structured Streaming, Delta Live Tables (Lakeflow Declarative Pipelines) and Azure Data Factory.
- Implement incremental processing, CDC, SCD Type 1/2, MERGE patterns, schema evolution and late-arriving data handling.
- Set data quality rules and expectations at each layer, with quarantine and alerting for bad records.

Platform architecture and scalability
- Architect the Azure Databricks environment: workspaces, dev/test/prod separation, VNet injection, Private Link, ADLS Gen2 storage layout and naming standards.
- Design for scale across data volume, number of pipelines, teams and concurrent users.
- Define cluster policies, compute types (job, all-purpose, serverless, SQL warehouses) and orchestration with Databricks Workflows.
- Build reusable frameworks: metadata-driven ingestion, common libraries, templates and logging.

Performance optimisation
- Tune Spark and Delta workloads: partitioning, liquid clustering, Z-ORDER, OPTIMIZE/VACUUM, file sizing,



caching, AQE and Photon.
- Diagnose and fix skew, shuffle, spill and small-file problems using the Spark UI and query profiles.
- Set and meet SLAs for pipeline run time and data freshness.

Cost optimisation (FinOps)
- Right-size clusters, use autoscaling, auto-termination, spot instances and serverless where it lowers cost.
- Enforce cost controls through cluster policies, tagging and budgets.
- Track DBU and Azure spend with system tables and dashboards, and report savings.

Governance, security and standards
- Implement Unity Catalog: metastore, catalogs/schemas per layer and domain, external locations, storage credentials.
- Set up role-based access, row- and column-level security, data masking and PII handling.
- Enable lineage, auditing, data classification and integration with Microsoft Purview.
- Manage secrets with Azure Key Vault and identities with Entra ID service principals and managed identities.
- Define and document engineering standards: coding guidelines, naming conventions, review process and data contracts.

DevOps and delivery
- Build CI/CD with Azure DevOps or GitHub Actions and Databricks Asset Bundles.
- Provision infrastructure as code with Terraform or Bicep.
- Add unit and integration tests, monitoring and alerting for pipelines.

Leadership
- Act as technical lead: mentor engineers, review code and design, and set best practices.
- Work with architects, analysts,



data scientists and business stakeholders to turn requirements into data products.

Required skills and experience
- 7+ years in data engineering, including 3+ years building production workloads on Azure Databricks.
- Proven track record of building a Medallion architecture from the ground up — be ready to walk through one you designed end to end.
- Expert in PySpark and Spark SQL; solid Python and SQL.
- Deep knowledge of Delta Lake: ACID transactions, time travel, MERGE, CDF, OPTIMIZE, liquid clustering.
- Hands-on with Unity Catalog and data governance: access control, lineage, auditing, PII protection.
- Strong Azure stack: ADLS Gen2, Azure Data Factory, Key Vault, Entra ID, Event Hubs, Azure Monitor, networking basics (VNet, Private Endpoints).
- Proven results in Spark performance tuning and cloud cost optimisation, with numbers to show for it.
- Data modelling: dimensional modelling (Kimball), Data Vault awareness, SCD patterns.
- CI/CD and IaC: Git, Azure DevOps or GitHub Actions, Databricks Asset Bundles, Terraform.
- Clear communicator who can explain design trade-offs to both engineers and business stakeholders.

Nice to have
- Certifications: Databricks Certified Data Engineer Professional, Microsoft Certified: Azure Data Engineer Associate (DP-203) or Fabric Data Engineer (DP-700), Databricks Lakehouse Platform Architect.
- Experience migrating from on-prem (SQL Server, Hadoop, Informatica, SSIS) or Synapse to Databricks.
- Databricks SQL, Lakehouse Monitoring, MLflow or Feature Store exposure.
- dbt on Databricks, Great Expectations, or similar data quality tools.
- Power BI integration and semantic model design on Gold data.
- Experience with data mesh or domain-oriented data products.

-

📌 Data Analyst, data engineers, data specialist (Pune)
🏢 Diacto Technologies
📍 Pune