21 Nov
|
Abh Consultants
|
India
21 Nov
Abh Consultants
India
Job Type: Freelance / Part-Time / Contract
Engagement: Approximately 2 hours per day
Duration: Ongoing, based on project requirements
Work Mode: Remote
Experience Required: 7+ years
Opportunity: Suitable for experienced professionals seeking additional freelance income or part-time consulting opportunities alongside their existing employment.
Fabric Data Engineer — Freelance / Part-Time Technical Support
We are seeking professionals with strong hands-on experience in Microsoft Fabric, Snowflake, Large-Scale Legacy Infrastructure Migration (1,000+ Tables), Enterprise Data Modeling, Metadata-Driven ETL/ELT Pipelines, Multi-Server/API Integration, and Production Data Operations.
1. Microsoft Fabric & Core Data Platforms
- Microsoft Fabric Unified Ecosystem (OneLake, Lakehouse, Fabric Data Warehouse, SQL Analytics Endpoint)
- Microsoft Fabric Data Factory (Data Pipelines, Copy Activity, Dataflows Gen2 with Power Query M)
- Microsoft Fabric Data Engineering (Spark Notebooks, PySpark, Spark SQL, Spark Job Definitions)
- Snowflake Cloud Data Platform (Snowpipe, Dynamic Tables, Snowpark, Streams & Tasks, Virtual Warehouses)
- Legacy On-Premises & Enterprise Databases (SQL Server, Oracle, Teradata, DB2, PostgreSQL, MySQL)
- OneLake Shortcuts & Database Mirroring (ADLS Gen2, AWS S3, Google Cloud Storage, Dataverse, Snowflake Iceberg)
2. Medallion Architecture Layers (Landing, Bronze, Silver, Gold & Semantic) Strong hands-on expertise designing, building, and governing multi-hop data layers across Microsoft Fabric and Snowflake:
- Landing / Raw File Zone – Ingesting raw structured, semi-structured, and unstructured files (Parquet, Avro, JSON, XML, CSV, Excel, API payloads) into OneLake Files / cloud staging areas with folder partitioning and retention policies
- Bronze Layer (Raw Immutable History) – Building append-only raw Delta Parquet tables in Fabric Lakehouse and Snowflake Raw schemas with schema-on-read, audit metadata columns (_ingest_timestamp, _source_system, _batch_id, _file_name), schema drift capture, and full historical replayability
- Silver Layer (Cleansed, Conformed & Deduplicated) – Implementing standardized Delta tables featuring data type casting, null/anomaly handling, deduplication, business key harmonization, PII masking/tokenization, cross-source entity resolution, and Slowly Changing Dimensions (SCD Type 1 & Type 2)
- Gold Layer (Curated Analytical & Business Marts) – Architecting dimensional models (Kimball Star/Snowflake schemas—Fact and Dimension tables), Data Vault 2.0 business vaults, and pre-aggregated KPI tables in Fabric Data Warehouse and Snowflake Marts
- Semantic & Consumption Layer (Direct Lake & Serving) – Designing Power BI Direct Lake semantic models over OneLake Delta tables (eliminating import/refresh latency), DAX measures, Row-Level Security (RLS), and governed data products for BI and AI/ML workloads
3. Large-Scale Legacy Migration & Multi-Source Integration
- Metadata-Driven ETL/ELT Frameworks – Parameterized, control-table driven Fabric pipelines built to ingest, track, and migrate thousands of legacy tables dynamically
- Multi-Server & Legacy RDBMS Extraction – Connecting to SQL Server, Oracle, Teradata, Netezza, Sybase, DB2, and legacy file servers via On-Premises Data Gateways, VNet Data Gateways, and ODBC/JDBC
- Multi-Format File & API Ingestion – Integrating REST/GraphQL/SOAP APIs, SFTP/SMB servers, Parquet, Avro, ORC, JSON, XML, CSV, Multi-Sheet Excel, and Fixed-Width flat files
- Change Data Capture (CDC) & Incremental Loads – Watermarking, SQL Change Tracking, Fabric Database Mirroring, Snowflake Streams, and Debezium log-based CDC
- Legacy ETL Reverse-Engineering – Translating legacy SSIS packages, Informatica mappings, and complex stored procedures into modular PySpark notebooks, Fabric pipelines, and T-SQL/Snowflake SQL
4. Technical, Programming & Performance Optimization Knowledge Practical understanding of relevant Microsoft Fabric and Snowflake technical concepts, such as:
- Advanced SQL and cross-dialect translation (T-SQL, PL/SQL, Teradata SQL to Spark SQL, Fabric Warehouse T-SQL, and Snowflake SQL)
- Python and PySpark programming for reusable transformation libraries, dynamic schema inference, and automated ingestion frameworks
- Storage and query performance tuning (Delta Lake V-Order optimization, Z-Ordering, Liquid Clustering, partitioning, file compaction/OPTIMIZE/VACUUM, and Snowflake micro-partition clustering)
- Hybrid networking and security (On-Premises Data Gateways, Managed Private Endpoints, Service Principals, Managed Identities, OAuth 2.0, Key Vault integration)
- External API integration patterns (cursor/offset pagination, rate-limit throttling, exponential backoff retries, and nested JSON/XML flattening)
- Data governance, lineage, and access control (Microsoft Purview integration, sensitivity labels, Workspace RBAC, Row-Level Security [RLS], Column-Level Security [CLS], and Dynamic Data Masking)
- CI/CD and DataOps automation (Fabric Git Integration with Azure DevOps/GitHub, Deployment Pipelines across Dev/Test/Prod, Variable Libraries, and parameterized connections) Deep expertise in all these technologies is not mandatory; working knowledge and relevant integration experience are expected.
5. Data Modeling, Cutover & Production Support
- Source-to-Target Data Mapping (STTM), Entity-Relationship Diagrams (ERDs), and data dictionaries across thousands of legacy tables
- Legacy dependency analysis, wave-based migration grouping, and cold/archival table segregation
- Automated pre- and post-migration data reconciliation (row counts, financial/numeric aggregates, null checks, and column-level hash checksums)
- System Integration Testing (SIT) and User Acceptance Testing (UAT) with downstream Power BI and analytics teams
- Phased wave cutover, parallel-run validation, and Go-Live activities
- Hypercare and post-migration production stabilization support
- Production pipeline monitoring, SLA incident management, and Root Cause Analysis (RCA)
- Capacity and compute cost optimization (Microsoft Fabric Capacity Units [F-SKU CUs], Spark autoscale/concurrency tuning, and Snowflake credit management)
6. Additional Microsoft Fabric, Snowflake & Cloud Technologies (Preferred) Experience in any of the following will be an added advantage:
- Microsoft Fabric Real-Time Intelligence (Eventstreams, Eventhouse / KQL Databases, Data Activator)
- Power BI Enterprise Modeling (Direct Lake Mode, Composite Models, DAX Studio, Tabular Editor)
- dbt (Data Build Tool) Core / Cloud for SQL transformation workflows
- Apache Airflow / Azure Data Factory (ADF) / ADLS Gen2
- Legacy ETL Platforms (SSIS, Informatica PowerCenter/IICS, Talend, DataStage)
- Automated Data Quality & Observability Frameworks (Excellent Expectations, Soda, Monte Carlo)
- Terraform / Azure Bicep for automated Fabric workspace and cloud infrastructure provisioning
- Certifications such as Microsoft Certified: Fabric Data Engineer Associate (DP-700), Azure Data Engineer Associate (DP-203), or SnowPro Core / Architect
Candidate Requirements
- Minimum 8 years of professional Data Engineering, Data Warehousing, and ETL/ELT consulting experience.
- Strong hands-on experience designing and implementing Medallion Architecture (Landing, Bronze, Silver, Gold, and Semantic layers) in Microsoft Fabric and/or Snowflake.
- Proven track record migrating large-scale legacy systems (hundreds to thousands of tables) into modern cloud lakehouse/warehouse environments.
- Deep expertise in enterprise data modeling (Dimensional Kimball, Data Vault 2.0) and metadata-driven ETL/ELT pipeline engineering.
- Practical experience integrating multiple servers, relational databases, REST/SOAP APIs, and diverse file formats.
- Strong command of SQL, Python/PySpark, Delta Lake optimization (V-Order/Z-Order), and hybrid gateway connectivity.
- Experience in end-to-end data migration, automated reconciliation, cutover validation, or production data support projects.
- Strong troubleshooting, root-cause analysis, and compute/query performance tuning abilities.
- Ability to explain complex Medallion data modeling and migration architectures through real-world project scenarios.
- Excellent English communication skills.
- Experience mentoring data engineers or conducting technical interviews is an advantage.
- Comfortable supporting multiple Microsoft Fabric, Snowflake, and Data Engineering-related job positions within their areas of expertise.
Pay: ₹20,000.00 - ₹30,000.00 per month
Benefits:
- Flexible schedule
- Work from home
Experience:
- Azure Data Factory: 5 years (Required)
- Azure Databricks: 5 years (Required)
- Azure Data Lake Storage (ADLS Gen2): 5 years (Required)
- PySpark: 5 years (Preferred)
- Power BI: 3 years (Preferred)
- Snowflake: 4 years (Preferred)
- Data Engineer: 7 years (Required)
- Microsoft Fabric: 3 years (Required)
- OneLake: 3 years (Required)
Work Location: Remote
📌 Microsoft Fabric Data Engineer — Freelance / Part-Time Technical Support (India)
🏢 Abh Consultants
📍 India