Job Description

Data professional with experience in building and validating end-to-end data pipelines across Azure Data Factory, Databricks, and ADLS. Skilled in data quality assurance, automation using Python and PySpark, and integrating tests into CI/CD workflows. Adept at large-scale data validation, monitoring data health, and ensuring reliable batch and streaming processes, with additional expertise in developing scalable BI dashboards and reporting solutions.

Design and implement end-to-end data validation strategies for pipelines built on Azure Data Factory, Databricks (Delta Lake), and ADLS
Perform source-to-target reconciliation across ingestion, transformation, and consumption layers
Implement robust checks for Data Quality Dimensions
Build scalable data testing frameworks using python, pyspark and positive knowledge of automation tools like playwright & pyspark based automation testing.
Integrate automated data tests into CI/CD pipelines (Azure DevOps / GitHub Actions)
Implement data quality-as-code practices using tools like Soda, dbt etc.
Validate Spark transformations,



Delta Lake tables, and streaming pipelines
Perform large-scale data validation using PySpark and SQL
Optimize validation logic for high-volume datasets
Ensure correctness of: Batch and streaming jobs, Incremental loads (CDC pipelines), Slowly changing dimensions (SCD)
Validate data movement, orchestration workflows, and failure handling in Azure Data factory & Azure Databricks services
Define and implement data quality SLAs and KPIs
Build dashboards to track data health and pipeline reliability
Proactively identify anomalies and data drifts
Implement alerting mechanisms for data failures
Design, develop, and optimize interactive dashboards and reports using BI tools (e.g., Power BI, Tableau, Looker).
Translate business requirements into technical reporting solutions.
Identify bottlenecks and improve report/query performance.
Implement best practices for dashboard design and scalability.
E

📌 Associate Principal Engineer (Bengaluru)
🏢 Wesco
📍 Bengaluru