- Design, develop, test, deploy and maintain large-scale data pipelines using AWS services such as S3, Lambda, Step Functions.
- Collaborate with cross-functional teams to gather requirements and design solutions for complex data processing needs.
- Develop high-quality code in Python using PySpark, Pandas, and other relevant libraries to process large datasets.
- Troubleshoot issues related to data pipeline failures or performance problems.
Job Requirements :
- 6-12 years of experience in a similar role as a Data Engineer with expertise in AWS technologies.
- Robust proficiency in Python programming language with experience working with popular libraries like Pandas and PySpark.
- Experience designing scalable architectures for big-data processing using Data Bricks or similar tools.