06 Sep
|
TalentOla
|
Bengaluru
06 Sep
TalentOla
Bengaluru
Job Summary
We are seeking a highly experienced Data Engineer (6-12 years) with expertise in Snowflake, DBT, Apache Airflow, and StreamSets, and strong hands-on experience in designing enterprise-grade ETL/ELT, data migration, and multi-source ingestion frameworks within the Life Sciences domain.
Key Responsibilities
1. Snowflake Architecture & Enterprise Data Platform Design
· Lead architecture and implementation of scalable Snowflake data platforms:
o Multi-layered architecture (Landing → Raw → Staging → Curated → Data Marts)
· Develop secure cross-account data sharing strategies.
· Implement:
o Snowpipe for automated ingestion
o Streams & Tasks for CDC-based incremental processing
o Time Travel & Zero-copy cloning for setting management
· Implement data masking, row-level security, and RBAC frameworks.
· Optimize storage, partitioning (micro-partition pruning), and query performance.
2. Data Migration & Modernization
· Participate in end-to-end data migration initiatives including:
o Legacy data warehouse (Teradata, Oracle, SQL Server, Netezza) to Snowflake
o On-prem to cloud modernization programs
o Source system analysis and profiling
o Data quality assessment and remediation planning
o Schema conversion and transformation mapping
o Incremental migration strategies
o Parallel-run validation strategies
· Perform reconciliation and data validation between legacy and target systems.
· Develop automated validation scripts using SQL and DBT tests.
· Support cutover planning and production readiness.
3. Data Ingestion & Multi-Source Integration
Design and implement ingestion frameworks for structured, semi-structured, and unstructured data from multiple enterprise systems:
Structured Sources
· Oracle, SQL Server, SAP, PostgreSQL
· Clinical systems (EDC, CDMS, CTMS)
· Regulatory systems (RIM)
· Commercial systems (CRM, ERP)
Semi-Structured Sources
· JSON, XML, Avro files
· API responses
· External vendor feeds
4. DBT – Enterpris
📌 Snowflake DBT(Neha) (Bengaluru)
🏢 TalentOla
📍 Bengaluru