Senior Databricks Data Quality Engineer

WORK FROM HOME (US/UK SHIFT)

Builds the reusable engine at the centre of the proposal and configures the native monitoring that replaces much of the estate. This is framework engineering rather than pipeline building — the output is a system that generates the migration, not code written rule by rule.

KEY RESPONSIBILITIES:

Build the config-driven framework: rule configuration repository, routing engine, alias resolver, code generator and results model.

Design the configuration schema so that adding a rule after handover is a config row rather than a code change.

Generate, deploy and support the DQ notebooks for those rules that require generated SQL.

Build the alias resolution layer that translates Collibra @dataset references into fully-qualified Unity Catalog paths.

Configure Databricks Lakehouse Monitoring across the in-scope tables, including slicing expressions that replace brand-partitioned rules.

Decompose Collibra adaptive rules into metric, scope, tolerance band and score weight; lift manual-tier bands from the export and re-derive model-derived bands from Delta time travel.





Warm monitor baselines from table history so monitors go live already informed rather than in a cold learning state.

Wire load-triggered execution into existing Databricks Workflows and establish CI/CD through Asset Bundles.

REQUIRED QUALIFICATIONS:

6+ years on Databricks and Spark with strong PySpark and SQL.

Demonstrable experience building reusable frameworks or internal platforms consumed by other engineers — not only delivering pipelines.

Production experience with Delta Lake, Unity Catalog, Databricks Workflows and Asset Bundles.

Hands-on Databricks Lakehouse Monitoring, or comparable data-observability tooling such as Monte Carlo, Anomalo or Soda.

Comfortable with the statistics behind behavioural monitoring — drift, baselines, tolerance bands, percentiles and sensitivity tuning.

Writes code a client team will own and extend after handover — clarity, structure and documentation weigh as heavily as function.

PREFERRED:

Exposure to Collibra DQ or OwlDQ behavioural analytics.

Experience with code generation, templating or metadata-driven pipeline patterns.

Exposure to open-source DQ frameworks such as Great Expectations, DQX or dbt tests.

📌 Senior Databricks Data Quality Engineer (India)
🏢 Quantum Integrators
📍 India