Data Engineer: Microsoft Fabric, Batch and Streaming Pipelines

Experience: 5+ years (adjust to the client's level) | Hours: Full time, 40 hrs/week, aligned to client's team time zone

Role Overview

Join a product-focused team to build and run the data pipelines behind customer-facing applications and analytics. You'll own your pipelines from first design through production support, with a focus on Microsoft Fabric.

Key Responsibilities
Develop robust batch and streaming pipelines that pull data from APIs, databases, event streams, and external vendor systems.
Create reusable patterns for ingesting, transforming, and storing data, and model it into consistent, trusted datasets.
Build and troubleshoot workflows in Microsoft Fabric Notebooks and move them into production.
Put automated testing, version control, and CI/CD around every pipeline you ship.
Set up data quality checks, alerting, and monitoring so issues surface before users notice them.
Tune queries, pipelines, and storage to keep runtime and cloud spend under control.
Maintain clear documentation of data flows, lineage, and metadata.




Work within agile ceremonies (planning, standups, retros) alongside product managers, developers, and analysts.
Follow data privacy and security standards, and help mentor junior engineers.

Mandatory Skills
Programming: Python, Java, or Scala for pipeline development
Data Frameworks: Hands-on with at least two of Spark, Kafka, Flink, dbt
Databases: Robust SQL plus working knowledge of NoSQL; solid data modeling fundamentals
Platform: Microsoft Fabric Notebooks, used in real project work
Containers & Orchestration: Docker and Kubernetes
Monitoring: Experience with observability tooling; Datadog strongly preferred
Performance: Proven SQL and pipeline optimization for speed and cost
Valuable to have: Fabric Lakehouse, Warehouse, Data Pipelines, Dataflows Gen2, Semantic Models; Azure or AWS data services; CI/CD tools (GitHub Actions, Jenkins, Azure DevOps); use of generative or agentic AI tools to speed up engineering work

📌 Data Engineer Hyderabad
🏢 Virtues R u0026 D
📍 Hyderabad