- Design, develop, and maintain large-scale data pipelines using Azure Data Factory (ADF) to extract, transform, and load data from various sources into Azure Databricks.
- Collaborate with cross-functional teams to gather requirements for data warehousing solutions and implement ETL processes to meet business needs.
- Develop complex SQL queries to optimize database performance and troubleshoot issues in existing databases.
- Ensure high availability of data systems by implementing monitoring tools and performing regular maintenance tasks.
Job Requirements :
- 6-9 years of experience in designing and developing large-scale data pipelines using ADF or similar technologies.
- Solid understanding of Azure Databricks platform including its features such as Spark clusters, workloads, and security concepts.
- Proficiency in writing complex SQL queries for querying relational databases like SQL Server or PostgreSQL.