18 Nov
|
Black Piano
|
India
Role: Senior Data Engineer (data architecture) works autonomously, leads local design and guides other engineers- platforms: Microsoft Fabric, Databricks, Microsoft Purview, Profisee
Role summary: A senior data engineer who combines hands-on PySpark delivery with data architecture input: leading local design, influencing design analysis, and owning data cataloguing and master data standards across Microsoft Purview and Profisee.
Activities & outputs:
Data architecture & design
- Lead local solution design for assigned data flows and influence wider design analysis, including Source-to-Target (S2T) mappings and medallion layer patterns
- Catalogue data assets in Microsoft Purview: metadata, classifications, business glossary terms and lineage
- Work with Profisee MDM: master data models, matching and survivorship rules, and integration of golden records with the lakehouse
- Challenge and improve designs, raising risks, trade-offs and dependencies early
Engineering & delivery:
- Deliver high-quality PySpark development in Fabric / Databricks notebooks within an existing Bronze–Silver–Gold architecture
- Extend and optimise existing ETL/ELT pipelines and downstream exports (batch, real - time and on-demand)
- Own data quality, performance tuning and issue resolution for assigned flows, applying modular, maintainable, version-controlled code
- Deliver against a high-volume backlog (200+ flows), prioritising critical milestones (e.g. June critical path) and adapting to changing requirements
- Collaboration & ways of working
- Act as a senior point of contact for technical teams and business stakeholders, communicating progress, blockers and dependencies clearly
- Guide and review the work of other engineers in the distributed/offshore team
- Follow client data policies, governance and security requirements; maintain documentation and traceability
Skills & experience- Essential:
- Microsoft Purview: data cataloguing, source scanning and classification, business glossary, lineage and data products
- Profisee MDM: entity modelling, match/merge and survivorship rules, data quality rules, stewardship workflows, and integration with Fabric / Azure Data Factory
- Data engineering: solid PySpark and SQL in Fabric / Databricks notebooks, with proven delivery in a medallion (Bronze–Silver–Gold) architecture
- Data design: data modelling and Source-to-Target mapping; turning business requirements into practical designs and leading local design decisions
- Engineering practice: data quality testing, performance tuning, Git version control and CI/CD
- Seniority: track record of working autonomously, mentoring engineers and engaging confidently with business stakeholders
Desirable:
- Certifications: Microsoft Fabric Data Engineer (DP-700) or Fabric Analytics Engineer (DP-600)
- Wider tooling: other MDM or catalogue tools, e.g. CluedIn, Informatica, Collibra
- Semantic layer: understanding of Power BI semantic models and Fabric IQ, and how mastered data feeds them
- Context: experience in regulated or data-intensive sectors, ideally utilities
📌 Senior Data Engineer (India)
🏢 Black Piano
📍 India