We are seeking a highly experienced Data Engineer (6-12 years) with expertise in Snowflake, DBT, Apache Airflow, and StreamSets, and strong hands-on experience in designing enterprise-grade ETL/ELT, data migration, and multi-source ingestion frameworks within the Life Sciences domain.
Key Responsibilities
1. Snowflake Architecture & Enterprise Data Platform Design
· Lead architecture and implementation of scalable Snowflake data platforms:
o Multi-layered architecture (Landing → Raw → Staging → Curated → Data Marts)
· Develop secure cross-account data sharing strategies.
· Implement:
o Snowpipe for automated ingestion
o Streams & Tasks for CDC-based incremental processing
o Time Travel & Zero-copy cloning for workplace management
· Implement data masking, row-level security, and RBAC frameworks.
· Optimize storage, partitioning (micro-partition pruning), and query performance.
2. Data Migration & Modernization
· Participate in end-to-end data migration initiatives including:
o Legacy data warehouse (Teradata, Oracle, SQL Server, Netezza) to Snowflake
o On-prem to cloud modernization programs
o Source system analysis and profiling
o Data quality assessment and remediation planning
o Schema conversion and transformation mapping
o Incremental migration strategies
o Parallel-run validation strategies
· Perform reconciliation and data validation between legacy and target systems.
· Develop automated validation scripts using SQL and DBT tests.
· Support cutover planning and production readiness.
3. Data Ingestion & Multi-Source Integration
Design and implement ingestion frameworks for structured, semi-structured, and unstructured data from multiple enterprise systems: