Job Title : Senior Data Engineer
Experience:

6+ Years
Location:

Chennai, Pune, Bengaluru, Hyderabad
Contract:

6 Months
Notice Period :

Immediate Joiners

Job Overview

We are seeking a hands-on Data Engineer to develop, optimize, and maintain automated data pipelines supporting data governance and analytics initiatives. This role will focus on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration, primarily using

Azure Databricks ,

Azure Data Lake , and

Microsoft Purview .

Key Responsibilities

Pipeline Development

– Design, build, and deploy robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.
Granular Data Quality Capture

– Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type)



to support drill-down from dashboards into specific records and enable targeted remediation.
Governance Metrics Automation

– Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
Microsoft Purview Integration

– Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
Data Retention & Policy Enforcement

– Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
Job & Query Monitoring

– Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
Metadata Storage & Optimizatio n – Maintain curated Delta tables for governance metrics, structured for productive dashboard consumption.
Testing & Troubleshooting

–

📌 Senior Data Engineer (Mumbai)
🏢 RapidBrains
📍 Mumbai