We are seeking an experienced Azure Platform Engineer (Databricks) to design, manage, optimize, and secure enterprise-scale Databricks environments on Azure. The ideal candidate will have robust expertise in Azure infrastructure provisioning, Databricks administration, Terraform automation, security compliance, and platform troubleshooting.
Key Responsibilities
Platform Design
Define and implement best practices for end-to-end Databricks platform architecture.
Collaborate with Databricks and internal teams to evaluate recent Databricks features, including private and public previews.
Engage with Databricks product teams on feature enhancements and platform improvements.
Platform Infrastructure
Create and manage Databricks workspaces (Premium, Standard, Serverless) and clusters with proper sizing strategies.
Decommission unused workspaces and optimize platform resources.
Support Delta Sharing initiatives and work with enterprise teams on connected data-sharing solutions.
User & Access Management
Create and manage security groups, users, and access controls.
Assign Unity Catalog permissions to respective groups and teams.
Manage Quantum Collaboration platform settings used as sandbox spaces for enterprise ideation and innovation.
Troubleshooting & Performance Optimization
Investigate and diagnose Databricks performance issues and platform errors.
Analyze Databricks logs and error messages to identify root causes.
Resolve issues related to cluster configurations, job failures, and notebook execution.
Optimize Databricks notebooks and jobs for better performance and scalability.
Coordinate with Databricks support teams for complex or unresolved issues.
Document troubleshooting procedures, known issues, and solutions.
Develop and test Databricks clusters to ensure platform stability and scalability.
Monitoring & Dashboarding
Design dashboards tailored for different user personas such as