Role summary

Build and operationalize the Fabric pipeline designed by the architect: ingestion from multiple sources, Spark-based transformation logic, Gold layer tables (both MLV and DirectQuery-served), and ongoing maintenance/optimization through go-live and stabilization. This is a sustained build-and-run role, following an existing architecture rather than designing it from scratch.

Key responsibilities

- Build and maintain Spark notebooks for BronzeSilver transformations, including parsing semi-structured (JSON) and NoSQL-sourced data

- Implement entity-resolution/join logic across sources (structured relational + semi-structured + NoSQL)

- Build and schedule Materialized Lake Views and/or DirectQuery views per the architecture's real-time/analytical split

- Set up and maintain Data Pipeline orchestration (ingestion, transform, refresh scheduling, dependency chains)

- Monitor and tune performance: table optimization/vacuuming, small-file compaction, Direct Lake troubleshooting

- Maintain and extend the mirroring/ingestion connections (Oracle mirroring, OFSC API polling, MongoDB connector) as data volume or requirements evolve





- Support integration testing with the AI agent layer (Fabric Data Agent and/or direct SQL/MCP tool queries)

- Document pipeline logic and hand over runbooks for BAU support

Must-have skills

- Robust hands-on PySpark and Spark SQL comfortable writing production transformation logic, not just tutorials

- Working experience with Microsoft Fabric: Lakehouse, notebooks, Data Pipelines, Delta tables

- Experience with Delta Lake concepts: schema evolution, optimize/vacuum, change data feed

- SQL proficiency (T-SQL for Warehouse/SQL analytics endpoint work)

- Experience with API-based data ingestion (REST APIs, JSON parsing, pagination/polling patterns)

- Comfortable working from an architecture document and translating design decisions into working code

Nice-to-have

- Materialized Lake Views experience specifically

- Experience with MongoDB or other NoSQL Spark connectors

- Power BI / semantic model exposure (helpful for testing Gold layer outputs, not core to the role)

- DP-203 or DP-700 (Fabric Data Engineer) certification.

📌 Microsoft Fabric Data Engineer (Hyderabad)
🏢 Qentelli
📍 Hyderabad