Job Summary

We are seeking a Senior AWS Data Engineer to lead the architectural design and implementation of a scalable, metadata-driven data ingestion and extraction framework. This role is focused on building highly flexible and customizable data processing pipelines that leverage the full AWS Cloud suite. You will be responsible for creating automated, reusable frameworks that handle diverse data scenarios while ensuring data integrity, high availability, and performance at scale.

Key Responsibilities Framework Development Lead the design and development of a scalable, metadata-driven data ingestion and extraction framework using Python, PySpark, and SQL.

Pipeline Engineering

Design and implementation of robust ETL/ELT pipelines leveraging AWS Glue, Lambda, and Step Functions to handle diverse data processing requirements. AWS Ecosystem Integration Seamlessly integrate various AWS services including S3, MSK (Kafka), Amazon Redshift, Athena, and RDS (Postgres, Oracle, MySQL) to build reliable end-to-end data flows. Infrastructure as Code Utilize AWS CDK or Terraform to automate the deployment and configuration of data infrastructure, ensuring consistency across environments.





Data Quality & Integrity Develop and maintain rigorous data validation processes and quality checks to safeguard the integrity of ingested data.

Operational Excellence

Implement robust error handling, centralized logging, monitoring, and alerting mechanisms to ensure the reliability of the entire data lifecycle.

Automation

Identify opportunities to automate repetitive tasks and build reusable modules to improve engineering efficiency and reduce manual intervention.

Required Technical Skills Core Languages Mastery of Python and PySpark for complex data processing, transformation logic, and automation scripting. AWS Data Services Extensive hands-on experience with AWS Glue, Redshift, S3, Lambda, and Step Functions. Messaging & Orchestration Proficiency in SQS, SNS, and building serverless workflows.

Database Expertise

Robust SQL skills with the ability to write complex queries from scratch.

Experience with both Relational (Aurora, Postgres, MySQL) and NoSQL databases. DevOps & Security Proficiency in AWS IAM for fine-grained access control and AWS CDK for infrastructure automation.

Data Ingestion

Proven experience with AWS Crawlers, schema evolution, and handling diverse file formats (Parquet, Avro, JSON).

📌 Gen AI (Hyderabad)
🏢 Virtusa
📍 Hyderabad