12 Oct
|
Virtusa
|
Chennai
Job Summary: We are seeking a Senior Data Engineer to design, build, and optimize our enterprise data platform leveraging Snowflake and AWS. You will be responsible for architecting scalable ETL/ELT pipelines, implementing advanced data modeling, and ensuring high-performance data delivery for analytics and machine learning. The ideal candidate is an expert in Snowflake’s cloud-native architecture and has a proven track record of managing complex data lifecycles in an AWS environment.
Key Responsibilities: Data Pipeline Engineering: Lead the design and implementation of reliable, metadata-driven ETL/ELT pipelines using dbt, Snowflake, and Python.
AWS Integration: Architect seamless data ingestion flows from diverse sources (APIs, RDBMS, SaaS) into Snowflake using AWS services such as S3, Lambda, and Kinesis/MSK.
Advanced Data Modeling: Design and optimize Star and Snowflake schemas, ensuring that data structures are high-performing, scalable, and optimized for analytical consumption.
Performance Engineering: Manage Snowflake compute resources (Virtual Warehouses), optimize micro-partitioning, and tune complex SQL queries to ensure maximum cost-efficiency and speed. Automation & CI/CD:
Drive engineering excellence by implementing version control (Git), automated testing via dbt, and continuous integration/deployment (CI/CD) workflows.
Orchestration: Develop and maintain sophisticated job schedules and dependencies using tools such as Airflow, Dagster, or Prefect. Data Governance & Quality: Implement automated data validation, monitoring, and alerting to ensure data integrity and compliance with enterprise standards (GDPR/HIPAA). Cross-functional Collaboration: Partner with Data Scientists and Business Stakeholders to translate complex requirements into technical data products and comprehensive documentation.
Technical Skills &
Qualifications: Snowflake Mastery: 3+ years of deep hands-on experience with Snowflake architecture, including clustering, zero-copy cloning, and secure data sharing. dbt (data build tool): Expert proficiency in dbt (Core or Cloud), including the use of macros, snapshots, testing, and documentation modules.
AWS Cloud Stack: Solid experience with AWS S3, IAM, Lambda, and Glue. Advanced SQL & Python: Mastery of SQL for complex transformations and Python for data engineering scripts and automation.
📌 Lead Software Engineer (Chennai)
🏢 Virtusa
📍 Chennai