We are looking for a AWS Data Engineer with 4-8 years of experience to design, develop, and maintain scalable data ingestion and transformation pipelines on AWS. The role focuses on building robust ETL solutions using PySpark and AWS Glue to support analytics and reporting use cases.
Key Responsibilities
- Design, develop, and maintain ETL pipelines using PySpark and AWS Glue.
- Build and optimize AWS Glue jobs (PySpark-based) for batch data processing.
- Ingest and transform structured and semi-structured data into data lakes or data warehouses.
- Implement data quality checks, validations, and reconciliation mechanisms.
- Optimize performance through partitioning, caching, and efficient Spark transformations.
- Write and optimize SQL for data validation and downstream analytical consumption.
- Prepare and maintain technical documentation including ETL mappings and LLDs.
- Support SIT/UAT activities and troubleshoot production issues as required.
Required Skills & Experience
- 45 years of hands-on experience in ETL / Data Engineering roles.
- Strong experience with PySpark for large-scale data processing.
- Hands-on experience with AWS Glue (Glue Jobs, Crawlers, Data Catalog).
- Solid SQL skills for querying, transformations, and validation.
- Good understanding of data warehousing and data lake concepts.
- Exposure to AWS services such as S3 and IAM.
- Experience with UNIX/Linux and basic shell scripting.
Good-to-Have Skills
- Exposure to AWS orchestration services such as Step Functions or Airflow.
- Experience in performance tuning and cost optimization on AWS.
- Exposure to CI/CD practices and version control tools like Git.
- Exposure to DataStage or migration projects is a plus.