Senior Data Engineer

Microsoft Fabric | Semantic Models | Canonical Data Architecture | AI & Machine Learning Enablement

DEPARTMENT

Data Architecture & Performance Engineering

OLE TYPE

Senior Individual Contributor

PRIMARY FOCUS

Fabric data engineering and modeling

CORE PLATFORMS

Microsoft Fabric, Azure Data Factory, Semantic and Canonical Models

EXPERIENCE

6+ years data engineering

PARTNERSHIPS

Architecture, AI, ML, data science, and business teams

Position Summary

We are seeking a highly skilled Senior Data Engineer to design, build, and optimize enterprise-scale data platforms that power analytics, artificial intelligence, machine learning, operational reporting, and executive decision-making.

This role is responsible for developing scalable data architectures within Microsoft Fabric, implementing enterprise semantic models, establishing canonical data models, and creating robust data integration solutions using Azure Data Factory and Fabric Data Factory. The successful candidate will create trusted, governed, and AI-ready data assets that support advanced analytics, data science experimentation, machine learning models, and intelligent business applications.

The ideal candidate possesses deep expertise in Microsoft Fabric, OneLake, Lakehouse architectures, semantic modeling, Azure Data Factory, enterprise data warehousing, dimensional modeling, and cloud-native data engineering. Experience integrating data platforms with data science workbenches, machine learning environments, Generative AI services, and intelligent applications is essential.

Key Responsibilities Microsoft Fabric Engineering

· Design, develop, deploy, and support enterprise data solutions using Microsoft Fabric, OneLake, Fabric Lakehouse, Fabric Warehouse, Fabric Data Factory, Fabric Data Engineering, Fabric Data Science, and Real-Time Intelligence.

· Implement scalable medallion architectures using Bronze, Silver, and Gold layers for controlled data ingestion, refinement, conformance, and consumption.

· Build data pipelines and reusable data products supporting operational analytics, advanced analytics, AI, and machine learning workloads.

· Optimize Fabric capacity utilization, storage, workload placement, refresh performance, concurrency, reliability, and operating cost.

· Establish source control, CI/CD, deployment, configuration, monitoring, observability, and support standards for Fabric environments.

Semantic Modeling and Data Product Development

· Design and maintain enterprise semantic models that provide governed, reusable, and business-friendly definitions of data, metrics, relationships, and analytical concepts.

· Develop reusable calculations, business rules, hierarchies, aggregations, measures, KPIs, and analytical frameworks.

· Build semantic layers that support reporting, analytics, machine learning, AI applications, and governed self-service data consumption.

· Optimize semantic models for scale, query performance, refresh efficiency, consistency, maintainability, security, and reuse.

· Establish semantic model standards, ownership, versioning, documentation, testing, certification, and lifecycle governance.

Canonical Data Modeling

· Design and maintain enterprise canonical data models that standardize shared business entities and data structures across source systems and business domains.

· Develop conceptual, logical, and physical data models aligned with enterprise terminology and business rules.

· Define reusable canonical entities, attributes, relationships, identifiers, event structures, and data contracts.

· Partner with architects, data owners, stewards, engineers, and business stakeholders to resolve conflicting definitions and establish trusted data assets.

· Reduce duplicate transformations and point-to-point mappings by creating reusable canonical integration and analytical structures.

Azure Data Factory and Enterprise Integration

· Architect, build, and maintain Azure Data Factory and Fabric Data Factory solutions for enterprise-scale ingestion, transformation, movement, and orchestration.

· Develop ETL and ELT pipelines, incremental processing, Change Data Capture, event-driven integrations, streaming patterns, and resilient recovery workflows.

· Integrate data from databases, files, APIs, SaaS applications, cloud services, on-premises platforms, and third-party systems.

· Create reusable ingestion frameworks, metadata-driven pipelines,



parameterized components, and integration accelerators.

· Implement secure connectivity, secrets management, data validation, error handling, logging, alerting, observability, and automated recovery.

AI, Machine Learning, and Data Science Enablement

· Create curated, governed, and AI-ready datasets for data scientists, machine learning engineers, AI developers, analysts, and intelligent applications.

· Integrate Microsoft Fabric and Azure data services with Azure Machine Learning, Fabric Data Science, MLflow, Azure AI Foundry, Azure OpenAI, model registries, and data science workbench environments.

· Design feature engineering and data preparation pipelines for model training, validation, testing, batch scoring, real-time inference, and model monitoring.

· Build data architectures supporting Generative AI, Retrieval-Augmented Generation, vector search, semantic retrieval, knowledge platforms, AI agents, and intelligent assistants.

· Enable secure, permission-aware, traceable, and governed access to enterprise data for AI and machine learning solutions.

· Collaborate with data science and AI teams to productionize experiments and establish repeatable data-to-model workflows.

Data Warehouse and Lakehouse Engineering

· Design and implement enterprise Lakehouse and Warehouse environments using modern cloud data architecture patterns.

· Build fact models, conformed dimensions, canonical business entities, curated data products, and aggregation layers.

· Apply Delta Lake practices for reliable, scalable, and maintainable data processing.

· Optimize storage layout, partitioning, file sizing, data access patterns, transformation placement, and workload performance.

· Support structured, semi-structured, batch, streaming, analytical, and AI-oriented data workloads.

Performance, Reliability, and Operational Excellence

· Optimize pipelines, semantic models, lakehouses, warehouses, notebooks, and integration workloads for performance and cost efficiency.

· Troubleshoot end-to-end data processing issues, including ingestion failures, query latency, transformation bottlenecks, resource contention, and data quality defects.

· Establish performance baselines, service objectives, health metrics, operational dashboards, alerting, and capacity indicators.

· Improve deployment quality through automated testing, data reconciliation, schema validation, observability, and controlled release practices.

Data Governance, Security, and Quality

· Implement metadata management, data lineage, cataloging, classification, retention, data quality, and stewardship controls.

· Apply least-privilege access, encryption, row-level controls, object-level controls, sensitivity labels, and auditability where required.

· Support integration with Microsoft Purview and enterprise governance processes.

· Design data quality rules, validation frameworks, reconciliation processes, exception handling, and measurable quality indicators.

· Ensure solutions comply with enterprise architecture, security, privacy, risk, and regulatory requirements.

Collaboration and Technical Leadership

· Partner with data architects, AI engineers, machine learning engineers, data scientists, application teams, product owners, governance teams, and business stakeholders.

· Translate business and analytical requirements into scalable data architecture, integration, modeling, and delivery solutions.

· Communicate technical designs, tradeoffs, risks, and recommendations clearly to technical and non-technical audiences.

· Participate in architecture reviews, establish engineering standards, mentor other engineers, and promote reusable delivery patterns.

· Create clear documentation for data models, pipelines, interfaces, data contracts, operational procedures, and support ownership.

Required Qualifications Education

· Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Data Science, Engineering, or a related discipline. Equivalent practical experience may be considered.

Experience





· 6+ years of professional data engineering, data platform engineering, or related experience.

· Demonstrated hands-on experience designing and delivering solutions with Microsoft Fabric.

· Experience developing and optimizing enterprise semantic models and canonical data models.

· Experience building production-grade Azure Data Factory and/or Fabric Data Factory solutions.

· Experience with enterprise data warehouses, lakehouses, data lakes, dimensional models, and cloud data integration.

· Experience enabling data science, machine learning, AI, or advanced analytics workloads.

Required Technical Skills

· Microsoft Fabric: OneLake, Lakehouse, Warehouse, Fabric Data Factory, Dataflows Gen2, Spark notebooks, Data Engineering, Data Science, and Real-Time Intelligence.

· Semantic and Canonical Modeling: Enterprise semantic models, canonical data models, conceptual/logical/physical modeling, dimensional modeling, star schemas, hierarchies, metrics, KPIs, business rules, and data contracts.

· Data Integration: Azure Data Factory, Fabric Data Factory, ETL/ELT, Change Data Capture, incremental processing, APIs, event-driven integration, streaming, orchestration, and metadata-driven pipelines.

· AI and Data Science: Azure Machine Learning, Azure AI Foundry, Azure OpenAI, MLflow, data science workbenches, feature engineering, model data preparation, RAG, vector data, and AI data pipelines.

· Programming: Advanced SQL and T-SQL, Python, PySpark, notebooks, scripting, and automation.

· Engineering Practices: Git, Azure DevOps, CI/CD, automated testing, monitoring, observability, source control, and controlled deployment.

· Data Platforms: Microsoft SQL Server, Azure SQL, PostgreSQL, Oracle, Snowflake, Databricks, Delta Lake, or comparable enterprise platforms.

Preferred Qualifications

· Microsoft Fabric Data Engineer, Fabric Analytics Engineer, Azure Data Engineer, Azure AI Engineer, or related certification.

· Experience integrating Microsoft Purview with enterprise metadata, lineage, classification, and governance processes.

· Experience supporting Generative AI, AI agents, vector search, machine learning lifecycle management, and production inference workloads.

· Experience with data contracts, master data, canonical integration patterns, and domain-oriented data products.

· Experience supporting regulated, mission-critical, high-volume, or globally distributed data environments.

Critical Competencies

· Strong analytical thinking, troubleshooting, and problem-solving skills.

· Explicit written and verbal communication with technical and non-technical stakeholders.

· Ability to connect business requirements to scalable data engineering and modeling solutions.

· Ownership mindset with attention to quality, security, performance, reliability, and maintainability.

· Collaborative approach and ability to work effectively across architecture, engineering, analytics, AI, data science, and business teams.

· Commitment to reusable engineering, documentation, knowledge sharing, and continuous improvement.

Success Measures

· Reliability, scalability, performance, and cost efficiency of Microsoft Fabric and Azure data solutions.

· Quality, reuse, consistency, and adoption of semantic models and canonical data assets.

· Pipeline success rates, data freshness, processing latency, recoverability, and operational supportability.

· Data quality, governance compliance, lineage coverage, security, and stakeholder trust.

· Speed and repeatability of data delivery for AI, machine learning, data science, and analytics use cases.

· Reduction in redundant data transformations, inconsistent definitions, and point-to-point integrations.

· Stakeholder satisfaction and measurable business value enabled by trusted enterprise data products.

Ideal Candidate Profile The ideal candidate is a senior data engineer who understands how enterprise data platforms enable analytics, machine learning, artificial intelligence, and business decision-making. They possess deep expertise in Microsoft Fabric, semantic and canonical modeling, Azure Data Factory, enterprise integration architectures, and data science enablement. They are equally comfortable working with data architects, AI engineers, machine learning engineers, data scientists, application teams, and business stakeholders to deliver trusted, scalable, reusable, and AI-ready enterprise data assets.

📌 Senior Data Engineer (Bengaluru)
🏢 Visionet Systems
📍 Bengaluru