- Design, develop, test, deploy and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, load data from various sources into Azure storage solutions.
- Collaborate with cross-functional teams to identify business requirements and design scalable data architectures that meet those needs.
- Develop complex SQL queries to optimize database performance and troubleshoot issues in Microsoft Azure databases.
- Implement machine learning models using PySpark on Azure Databricks for big data analytics.
Job Requirements :
- 5-10 years of experience in IT Services & Consulting industry with expertise in Microsoft Azure platform.
- Solid understanding of ADF workflows, triggers, connectors, datasets etc. .
- Proficiency in Python programming language with hands-on experience working with PySpark for big data processing.