We are seeking an experienced Azure Platform Engineer (Databricks) to design, manage, optimize, and secure enterprise-scale Databricks environments on Azure. The ideal candidate will have robust expertise in Azure infrastructure provisioning, Databricks administration, Terraform automation, security compliance, and platform troubleshooting.
Key Responsibilities
Platform Design
- Define and implement best practices for end-to-end Databricks platform architecture.
- Collaborate with Databricks and internal teams to evaluate new Databricks features, including private and public previews.
- Engage with Databricks product teams on feature enhancements and platform improvements.
Platform Infrastructure
- Create and manage Databricks workspaces (Premium, Standard, Serverless) and clusters with proper sizing strategies.
- Decommission unused workspaces and optimize platform resources.
- Support Delta Sharing initiatives and work with enterprise teams on connected data-sharing solutions.
User &
- Access Management
- Create and manage security groups, users, and access controls.
- Assign Unity Catalog permissions to respective groups and teams.
- Manage Quantum Collaboration platform environments used as sandbox spaces for enterprise ideation and innovation.
Troubleshooting &
- Performance Optimization
- Investigate and diagnose Databricks performance issues and platform errors.
- Analyze Databricks logs and error messages to identify root causes.
- Resolve issues related to cluster configurations, job failures, and notebook execution.
- Optimize Databricks notebooks and jobs for better performance and scalability.
- Coordinate with Databricks support teams for complex or unresolved issues.
- Document troubleshooting procedures, known issues, and solutions.
- Develop and test Databricks clusters to ensure platform stability and scalability.
Monitoring &
- Dashboarding
- Design dashboards tailored for different user personas such as data scientists, engineers, and administrators.
- Integrate Databricks APIs and monitoring tools to provide real-time operational metrics.
- Configure alerts for performance degradation and unexpected resource utilization.
- Ensure dashboards are user-friendly and provide actionable insights.
- Continuously improve dashboards based on feedback and evolving business requirements.
Security &
- Compliance
- Analyze findings from Databricks Security and Compliance reports.
- Apply necessary configurations and updates to remediate security vulnerabilities.
- Track and validate the effectiveness of implemented security controls.
- Collaborate with cybersecurity teams on enterprise security changes and governance initiatives.
Azure Infrastructure Provisioning &
- Automation
- Strong hands-on expertise in Terraform for Azure infrastructure provisioning and automation.
- Build and manage Infrastructure as Code (IaC) solutions for scalable Azure environments.
- Automate deployment and configuration processes across cloud infrastructure.
Required Skills
- Azure Databricks Administration
- Azure Cloud Infrastructure
- Terraform / Infrastructure as Code (IaC)
- Unity Catalog &
- Access Management
- Databricks Cluster Management
- Delta Sharing
- Monitoring &
- Alerting Tools
- Performance Tuning &
- Troubleshooting
- Security &
- Compliance Management
- CI/CD &
- Automation
- Strong problem-solving and communication skills
Preferred Qualifications
- Experience with enterprise-scale Databricks implementations.
- Knowledge of Azure networking and security best practices.
- Familiarity with DevOps and automation frameworks.
- Experience working with cross-functional enterprise teams.