- Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure storage solutions such as Blob Storage, Cosmos DB, Data Lake Storage, etc.
- Collaborate with cross-functional teams to gather requirements for data processing needs and design scalable architectures that meet business needs.
- Develop complex SQL queries to optimize database performance and troubleshoot issues in Azure Databricks.
- Ensure high availability of data systems by implementing redundancy measures and monitoring system health.