We are seeking an experienced Azure Platform Engineer (Databricks) to design, manage, optimize, and secure enterprise-scale Databricks environments on Azure. The ideal candidate will have robust expertise in Azure infrastructure provisioning, Databricks administration, Terraform automation, security compliance, and platform troubleshooting.
Key Responsibilities
Platform Design
- Define and implement best practices for end-to-end Databricks platform architecture.
- Collaborate with Databricks and internal teams to evaluate new Databricks features, including private and public previews.
- Engage with Databricks product teams on feature enhancements and platform improvements.
Platform Infrastructure
- Create and manage Databricks workspaces (Premium, Standard, Serverless) and clusters with proper sizing strategies.
- Decommission unused workspaces and optimize platform resources.
- Support Delta Sharing initiatives and work with enterprise teams on connected data-sharing solutions.
User & Access Management
- Create and manage security groups, users, and access controls.
- Assign Unity Catalog permissions to respective groups and teams.
- Manage Quantum Collaboration platform environments used as sandbox spaces for enterprise ideation and innovation.
Troubleshooting & Performance Optimization
- Investigate and diagnose Databricks performance issues and platform errors.
- Analyze Databricks logs and error messages to identify root causes.
- Resolve issues related to cluster configurations, job failures, and notebook execution.
- Optimize Databricks notebooks and jobs for better performance and scalability.
- Coordinate with Databricks support teams for complex or unresolved issues.
- Document troubleshooting procedures, known issues, and solutions.
- Develop and test Databricks clusters to ensure platform stability and scalability.
Monitoring & Dashboarding
- Design dashboards tailored for different user personas such as