- Design, develop, test, deploy and maintain large-scale data pipelines using Airflow to extract insights from Snowflake databases.
- Collaborate with cross-functional teams to identify business requirements and design scalable solutions for data processing and analysis.
- Develop complex SQL queries to optimize database performance and troubleshoot issues in real-time.
- Ensure high availability of the system by implementing monitoring tools and performing regular maintenance tasks.
Job Requirements :
- 7-9 years of experience in Data Engineering with expertise in Airflow, Python (PySpark), SQL, and Snowflake.
- Robust understanding of big data technologies such as Hadoop ecosystem components like HDFS, MapReduce, Spark Core etc. .
- Experience working on building scalable data warehouses/data lakes using various ETL tools like Informatica PowerCenter or similar technologies.