We are seeking an experienced AWS PySpark Developer with 5+ years of experience in building scalable data processing solutions and ETL pipelines on AWS. The ideal candidate should possess solid expertise in PySpark, Python, SQL, and AWS Data Engineering services with hands-on experience in developing and optimizing big data applications.
Key Responsibilities
Design, develop, and maintain large-scale data pipelines using PySpark.
Build and optimize ETL/ELT workflows on AWS platforms.
Develop data processing applications using Python and PySpark.
Work with AWS services including S3, EMR, Glue, and Airflow.
Analyze and optimize Spark jobs for performance and scalability.
Create reusable frameworks and components for data engineering solutions.
Collaborate with cross-functional teams to understand business and technical requirements.
Ensure data quality, reliability, and performance across data pipelines.
Support production deployments and troubleshoot data processing issues.
Required Skills
5+ years of overall IT experience.
3+ years of hands-on experience in PySpark development.
Robust proficiency in Python and SQL.
Hands-on experience with:
PySpark
AWS EMR
AWS S3
AWS Glue
Apache Airflow
Good understanding of ETL/Data Engineering concepts.
Experience in performance tuning and optimization of Spark applications.
Exposure to data warehousing and data lake architectures.
Experience with Git and CI/CD processes.
Preferred Skills
Basic understanding of Generative AI concepts.
Experience with Delta Lake, Iceberg, or similar technologies.
AWS Certification is an added advantage.
Solid analytical and problem-solving skills.