- Design, develop, test, deploy and maintain large-scale data pipelines using AWS services such as S3, Lambda, Step Functions.
- Collaborate with cross-functional teams to gather requirements for ETL processes and implement solutions that meet business needs.
- Develop complex SQL queries to extract insights from MySQL databases and perform analysis on large datasets using Python libraries like Pandas.
- Troubleshoot issues related to data processing workflows and provide recommendations for improvement.
Job Requirements :
- 3-7 years of experience in designing and developing ETL/ELT processes using various tools like PostgreSQL, MySQL, PySpark etc. .
- Robust understanding of AWS cloud platform including S3 buckets, Lambda functions, Step Functions etc. .
- Proficiency in writing complex SQL queries to extract insights from relational databases (MySQL).
- Experience working with big-data technologies like Hadoop ecosystem (Hive) is an added advantage.