- We are looking for an experienced AWS Glue Developer Data Engineer to design develop and optimize scalable ETL ELT data pipelines on AWS
- The candidate should have solid hands on experience with AWS Glue Python PySpark SQL data lake technologies and AWS data services
Key Responsibilities:
- Design develop and maintain ETL ELT pipelines using AWS Glue
- Develop AWS Glue jobs using PySpark and SQL for large scale data processing
- Build data ingestion and transformation pipelines across structured and semi structured data sources
- Configure and manage AWS Glue Data Catalog Jobs Workflows and Triggers
- Work with AWS services such as Amazon S3 Redshift and IAM
- Implement data transformations validations data quality checks and reconciliation processes
- Optimize Glue Spark jobs for performance scalability reliability and cost efficiency
- Implement incremental loads partitioning strategies error handling restart recovery and audit frameworks
- Troubleshoot ETL pipeline failures and production performance issues
- Implement appropriate security controls including IAM roles encryption and access management
- Support migration of legacy mainframe or on premises ETL workloads to the AWS cloud data platform
- Develop automated deployment processes using CI CD and Infrastructure as Code practices where applicable
- Collaborate with data architects analysts business teams and cloud platform teams to translate requirements into scalable data solutions
- Prepare technical design documents mapping specifications operational documentation and deployment runbook documentation
Additional Responsibilities:
- Technical Skills
- AWS Glue
- Python PySpark Apache Spark
- SQL
- Amazon S3
- Data Lake Cloud Data Warehouse concepts
- ETL ELT design and development
- Performance optimization and troubleshooting
- Experience
- 7 years of overall Data Engineering ETL development experience
- Strong hands on experience developing production grade AWS Glue Python and PySpark solutions
- Experience handling high volume datasets and complex transformations in AWS environments
- Preferred Certification
- AWS Certified Data Engineer Associate or other relevant AWS certification