Key Responsibilities

Pipeline Development & Automation

- Design, build, and maintain CI/CD pipelines to automate deployment of DQ rules and data services across environments.

- Optimize data pipelines and jobs for efficiency, scalability, and enterprise-grade reliability within Azure and Databricks.

- Implement best practices in version control, testing, and automated deployments.

Data Quality & Rule Development

- Develop, optimize, and maintain DQ rules in PySpark/Python for self-serve capabilities.

- Design and implement profiling frameworks for rule generation and automated remediation.

- Ensure DQ frameworks align with enterprise standards, governance, and audit requirements.

UI & Self-Serve Integration

- Collaborate with front-end teams (Node.js/Angular) to enable rule configuration, validation, and monitoring via a low-code UI.

- Develop APIs and services to expose DQ results and outputs for dashboards and self-service tools.

Collaboration & Governance

- Partner with Data Stewards, GPOs, and business owners to translate requirements into technical solutions.

- Contribute to BRDs, SRDs, and design reviews for DQ rules and pipelines.





- Provide inputs on governance, compliance, and change management processes.

- Support periodic reviews and continuous improvements of pipelines, rules, and dashboards.

Required Skills & Qualifications

- Strong programming skills in PySpark, Python, and SQL.
- Hands-on experience with Databricks (clusters, notebooks, Delta Lake).
- Experience designing and implementing CI/CD pipelines (Azure DevOps / GitHub Actions / Jenkins).
- Familiarity with Node.js & Angular for UI integration.
- Solid understanding of data governance, DQ frameworks, and MDM concepts.
- Robust analytical, problem-solving, and stakeholder engagement skills.
- Cloud expertise, preferably Azure Data Lake, Data Factory, and Synapse.

Education & Experience

- Bachelor’s or Master’s degree in Computer Science, Information Systems, or related field.
- 4–8 years of relevant experience in data engineering, CI/CD, and DQ rule development.
- Proven expertise in Databricks, Azure data services, and low-code self-serve frameworks.

📌 Data Engineer with DevOps (India)
🏢 Relevance Labs
📍 India