Role: Senior Databricks Data Quality Engineer

Location : Bangaloare & Nodia (2 Days onsite)

Shift Timing : UK Shift:

Candidate Availiiblity : Immediate Joiner

NOTE: We are specifically looking for engineers with experience in migrating from Collibra to Databricks . Also with experience in DQ migration framework generation .

Builds the reusable engine at the centre of the proposal and configures the native monitoring that replaces much of the estate. This is framework engineering rather than pipeline building — the output is a system that generates the migration, not code written rule by rule.

KEY RESPONSIBILITIES

- Build the config-driven framework: rule configuration repository, routing engine, alias resolver, code generator and results model.
- Design the configuration schema so that adding a rule after handover is a config row rather than a code change.
- Generate, deploy and support the DQ notebooks for those rules that require generated SQL.
- Build the alias resolution layer that translates Collibra @dataset references into fully-qualified Unity Catalog paths.
- Configure Databricks Lakehouse Monitoring across the in-scope tables, including slicing expressions that replace brand-partitioned rules.
- Decompose Collibra adaptive rules into metric, scope,



tolerance band and score weight; lift manual-tier bands from the export and re-derive model-derived bands from Delta time travel.
- Warm monitor baselines from table history so monitors go live already informed rather than in a cold learning state.
- Wire load-triggered execution into existing Databricks Workflows and establish CI/CD through Asset Bundles.

REQUIRED QUALIFICATIONS

- 8+ years on Databricks and Spark with robust PySpark and SQL.
- Demonstrable experience building reusable frameworks or internal platforms consumed by other engineers — not only delivering pipelines.
- Production experience with Delta Lake, Unity Catalog, Databricks Workflows and Asset Bundles.
- Hands-on Databricks Lakehouse Monitoring, or comparable data-observability tooling such as Monte Carlo, Anomalo or Soda.
- Comfortable with the statistics behind behavioural monitoring — drift, baselines, tolerance bands, percentiles and sensitivity tuning.
- Writes code a client team will own and extend after handover — clarity, structure and documentation weigh as heavily as function.

PREFERRED

- Exposure to Collibra DQ or OwlDQ behavioural analytics.
- Experience with code generation, templating or metadata-driven pipeline patterns.
- Exposure to open-source DQ frameworks such as Great Expectations, DQX or dbt tests.

Regards

Vibha Patel [email protected]

📌 Senior Databricks Data Quality Engineer (Bengaluru)
🏢 Quantum Integrators
📍 Bengaluru