13 Aug
|
Aliqan Services
|
India
13 Aug
Aliqan Services
India
Role Summary
We are seeking an experienced Data Engineer Microsoft Fabric with robust expertise in designing and developing enterprise-scale data solutions using Microsoft Fabric. The ideal candidate should have extensive hands-on experience with PySpark, SQL, Medallion Architecture, and Lakehouse design principles. This role involves building scalable data ingestion frameworks, developing production-grade ETL/ELT pipelines, and delivering high-quality data solutions for analytics and reporting.
Key Responsibilities
- Perform data analysis to understand source systems and business requirements.
- Translate business requirements into scalable data models and pipeline designs.
- Design, build, and maintain data ingestion pipelines from multiple data sources using batch and incremental (delta) loading techniques.
- Develop ingestion scripts for databases, flat files, APIs, and other enterprise data sources.
- Ingest and manage raw data in the Bronze layer while ensuring reliability, scalability, and traceability.
- Transform raw data into standardized Silver datasets through validation, deduplication, and conformance.
- Design and develop Gold layer models, including fact tables, dimension tables, and aggregated datasets for analytics.
- Build end-to-end ETL/ELT pipelines across Bronze, Silver, and Gold layers.
- Develop semantic models and reporting-ready datasets.
- Implement data quality rules and validation checks to ensure data completeness, accuracy, freshness, and reconciliation.
- Design robust error-handling mechanisms, including retry logic, logging, exception handling, and alerting.
- Develop self-healing pipelines using checkpointing, idempotent processing, auto-retry, and recovery mechanisms.
- Optimize pipeline performance for production-grade workloads.
- Monitor pipeline execution and proactively resolve failures and performance bottlenecks.
- Implement CI/CD pipelines and support deployments across Development, Test, and Production environments.
- De
📌 Data Engineer Microsoft Fabric (India)
🏢 Aliqan Services
📍 India