:

: Data Engineer – Databricks & Microsoft Fabric

Location : Chennai, India

Experience : 6 to 8 years

Role Summary

We are looking for an experienced Data Engineer with strong hands-on expertise in Databricks, Microsoft Fabric, PySpark, SQL, and cloud-based data engineering. The candidate will be responsible for designing, developing, and optimizing scalable data pipelines, lakehouse solutions, and analytics-ready data models.

The role requires robust experience in building end-to-end data engineering solutions, working with structured and semi-structured data, implementing data quality controls, and supporting enterprise reporting and analytics platforms.

Key Responsibilities

Data Engineering and Pipeline Development

- Design, build, and maintain scalable data pipelines using Databricks, PySpark, Spark SQL, and Microsoft Fabric.
- Develop batch and incremental data ingestion pipelines from multiple source systems.
- Build and optimize Bronze, Silver, and Gold layer data models using lakehouse architecture.
- Implement ELT/ETL workflows for data transformation, enrichment, validation, and publishing.
- Work with structured, semi-structured, and unstructured data formats such as CSV, Parquet, JSON, Delta, and XML.

Databricks Development

- Develop notebooks, jobs, workflows, and reusable components in Azure Databricks.
- Implement Delta Lake features such as schema evolution, merge/upsert, time travel, and optimized storage.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Implement partitioning, caching, indexing, and cluster optimization strategies.
- Troubleshoot job failures, performance bottlenecks, and data quality issues.

Microsoft Fabric Development

- Build data solutions using Microsoft Fabric Lakehouse, Warehouse, Data Factory, Pipelines, Notebooks, and OneLake.
- Develop and manage data pipelines in Fabric for ingestion, transformation, and orchestration.
- Work with Fabric SQL endpoints, semantic models, and Power BI integration.
- Support migration or modernization of existing data platforms into Microsoft Fabric.
- Implement reusable data engineering patterns and framework-based development in Fabric.

Data Quality, Governance, and Security

- Implement data validation, reconciliation, exception handling, and audit controls.
- Define and apply data quality rules including null checks, duplicate checks,



referential checks, and cross-field validations.
- Maintain data lineage, metadata, source-to-target mapping, and technical documentation.
- Ensure data pipelines comply with enterprise security, access control, and governance standards.
- Support integration with data governance tools such as Microsoft Purview, where applicable.

DevOps and Production Support

- Implement CI/CD practices for notebooks, pipelines, SQL scripts, and configuration files.
- Use Git-based version control and deployment processes across environments.
- Monitor production jobs and resolve incidents within agreed timelines.
- Prepare runbooks, deployment guides, operational support documents, and handover materials.
- Collaborate with architects, business analysts, data analysts, and reporting teams to deliver reliable data solutions.

Required Skills

Technical Skills

- Strong hands-on experience in Azure Databricks.
- Strong experience in Microsoft Fabric components such as Lakehouse, Warehouse, Data Factory, Pipelines, Notebooks, and OneLake.
- Proficiency in PySpark, Spark SQL, Python, and SQL.
- Strong knowledge of Delta Lake, lakehouse architecture, and medallion architecture.
- Experience with cloud storage and data platforms, preferably Azure Data Lake Storage, Azure SQL, Synapse, or Fabric OneLake.
- Experience in data ingestion from databases, APIs, files, SFTP, cloud storage, and streaming sources.
- Good understanding of data modeling, dimensional modeling, and analytics-ready data structures.
- Experience with performance tuning of Spark jobs and SQL queries.
- Experience in job scheduling, monitoring, logging, and error handling.
- Knowledge of CI/CD, Git, Azure DevOps, and deployment automation.

Preferred Skills

- Experience with Power BI and semantic model integration.
- Experience in migrating workloads from legacy ETL tools, Synapse, ADF, or Databricks to Microsoft Fabric.
- Knowledge of Microsoft Purview for data cataloging, lineage, and governance.




- Experience in building reusable data engineering frameworks.
- Exposure to real-time or near-real-time data processing.
- Azure certifications or Databricks certifications are preferred.

Roles and Responsibilities Summary

- Build and maintain scalable data pipelines using Databricks and Microsoft Fabric.
- Develop lakehouse solutions using Bronze, Silver, and Gold architecture.
- Perform data transformation, validation, reconciliation, and publishing.
- Optimize Spark workloads and SQL queries for performance.
- Implement data quality, audit, monitoring, and exception handling frameworks.
- Support deployment, production monitoring, incident resolution, and documentation.
- Collaborate with cross-functional teams to deliver enterprise data and analytics solutions.

Required Experience

- 6 to 8 years of overall experience in data engineering, ETL/ELT, or data platform development.
- At least 3+ years of hands-on experience in Databricks / PySpark.
- At least 1+ year of hands-on experience or strong working knowledge of Microsoft Fabric.
- Experience working in enterprise-scale data platforms and analytics projects.
- Experience in Agile delivery models and production support environments.

Educational Qualification

Bachelor’s degree in computer science, Information Technology, Engineering, Data Analytics, or a related discipline.

Good to Have Certifications

- Microsoft Certified: Fabric Analytics Engineer Associate
- Microsoft Certified: Azure Data Engineer Associate
- Databricks Certified Data Engineer Associate / Professional
- Microsoft Certified: Azure Fundamentals

Key Deliverables

- Production-ready data pipelines and notebooks.
- Optimized Databricks and Fabric workloads.
- Bronze, Silver, and Gold layer data models.
- Data quality and reconciliation reports.
- Source-to-target mapping and technical design documents.
- Deployment guides, runbooks, and support documentation.

Recruiter Name: Srinija Adapa

Recruiter Email ID: [email protected]

We transform. SAP® solutions into Value

NTT DATA Business Solutions is a fast-growing international IT company and one of the world’s leading SAP partners. We are a full service provider delivering everything from business consulting to implementation of SAP solutions, including hosting services and support.

📌 Data Engineer - Databricks & Microsoft FabricData Engineer - Databricks & Microsoft Fabric (Chennai)
🏢 NTT Data
📍 Chennai