Roles and Responsibilities

- Design, develop, and maintain ETL/ELT pipelines using Python and AWS Glue.
- Extract, transform, and load data from multiple sources into target systems.
- Develop data processing and transformation jobs using Python and PySpark.
- Use Amazon S3 for data storage, organization, and processing of structured and semi-structured data.
- Implement data cleansing, validation, transformation, and deduplication logic.
- Work with AWS Glue Data Catalog and crawlers to manage metadata.
- Write and optimize SQL queries for data extraction, transformation, and validation.
- Monitor ETL jobs, troubleshoot failures, and improve pipeline performance.
- Implement error handling, logging, and data quality checks.
- Collaborate with data engineers, analysts, and other stakeholders to understand data requirements.




- Maintain technical documentation and follow data engineering best practices.

Must-Have Skills

- Strong hands-on experience in ETL development and data pipeline implementation.
- Proficiency in Python programming and scripting.
- Mandatory experience with AWS Glue and Amazon S3.
- Experience with PySpark for large-scale data processing and transformations.
- Robust SQL skills, including joins, subqueries, aggregations, and query optimization.
- Experience working with CSV, JSON, and Parquet data formats.
- Understanding of data warehousing concepts and data quality practices.
- Experience in debugging, troubleshooting, and optimizing ETL pipelines.

📌 Aws Data Engineer (Hyderabad)
🏢 CGI
📍 Hyderabad