We are seeking a skilled
Azure Databricks Engineer
to design, develop, and optimize scalable data solutions on Microsoft Azure. The ideal candidate should have expertise in Azure Databricks, PySpark, Delta Lake, Azure Data Factory, and cloud-based data engineering architectures. The role involves building enterprise-grade data pipelines, implementing data transformation frameworks, and enabling advanced analytics and reporting solutions.
Key Responsibilities
Data Engineering & Pipeline Development
- Design and develop scalable ETL/ELT pipelines using Azure Databricks.
- Process large volumes of structured, semi-structured, and unstructured data.
- Develop and maintain PySpark and Spark SQL solutions.
- Build reusable data ingestion and transformation frameworks.
Azure Databricks Development
- Create and manage Databricks notebooks, workflows, and jobs.
- Implement Delta Lake for reliable and high-performance data processing.
- Configure and optimize Databricks clusters.
- Develop data solutions following Lakehouse architecture principles.
Azure Cloud Integration
- Integrate Databricks with Azure services such as:
- Azure Data Factory (ADF)
- Azure Data Lake Storage Gen2 (ADLS)
- Azure Synapse Analytics
- Azure SQL Database
- Azure Key Vault
- Event Hub