Data Discovery: Identify the SAP tables and fields used for existing Data Quality(DQ) rules & reports and locate the corresponding data in Databricks
Data Validation: Validate data across all the layers(bronze, silver,etc) in Databricks for completeness, structure and availability of required fields
DQ Rule Migration &
- BUILD: Select and build a set of existing DQ rules in Databricks as part of the PoC
PoC Testing &
- Reconciliation: Execute the rules in Databricks and compare results with the existing DQ results, investigating and resolving differences
End-to-End Rule Migration: Following the PoC, migrate / BUILD the agreed As-Is DQ rules to data bricks, including SQL logic, data mappings, rule outputs and required transformations