We are looking for an experienced AWS PySpark Data Engineer with robust hands-on expertise in building scalable data engineering solutions on AWS. The role will focus on developing and optimizing ETL/ELT pipelines, processing large-scale datasets using PySpark/Spark, and building cloud-native data solutions using AWS services.
Mandatory Skills
- 5+ years of overall experience
- PySpark / Apache Spark
- Python
- SQL
- AWS Glue
- Amazon S3
- AWS EMR
- AWS Lambda
- Amazon Redshift
- ETL/ELT development
- Data Lake/Data Warehouse concepts
- Large-scale/distributed data processing
Key Responsibilities
- Design, develop and maintain scalable ETL/ELT pipelines using PySpark.
- Build data ingestion and transformation frameworks on AWS.
- Process large volumes of structured,
semi-structured and unstructured data.
- Develop and optimize Spark/PySpark workloads.
- Build cloud-native Data Lake and Data Warehouse solutions.
- Implement data quality, validation, monitoring and governance.
- Monitor and troubleshoot production data pipelines.
- Participate in CI/CD, deployments and Agile delivery.
- Work closely with data architects, analytics teams and business stakeholders.