- Design, develop, test, deploy and maintain large-scale data pipelines using Azure Databricks.
- Collaborate with cross-functional teams to gather requirements and deliver high-quality solutions on time.
- Develop complex data processing workflows using PySpark and Azure Data Factory (ADF) to extract insights from various sources.
- Troubleshoot issues related to data quality, performance optimization, and scalability in the pipeline.
Job Requirements :
- 5-10 years of experience in designing and developing big data solutions using Azure Databricks.
- Robust expertise in working with PySpark for data processing tasks.
- Experience with ADF (Azure Data Factory) for orchestrating ETL processes.
- Proficiency in managing large datasets through efficient querying techniques.