Data Platform Engineer with a strong focus on Databricks and Azure to design, build, and operate a scalable and secure data platform. You will work at the intersection of infrastructure, data engineering, and cloud automation to establish an enterprise platform for analytics and AI workloads.
LONGDESCRIPTION section. 2 of 6.Section Title: Key Responsibilities
Key Responsibilities
- Automate the provisioning of Databricks and Azure platform resources using Terraform, adhering to Infrastructure-as-Code (IaC) best practices
- Design and develop reusable, modular Terraform components to enable scalable and maintainable deployments within the Databricks platform
- Implement and maintain CI/CD pipelines using Azure DevOps for automated infrastructure and platform deployments
- Provision current infrastructure using customized Databricks Terraform modules to support requests for recent catalogs, job policies, and clusters in a standardized, automated, and governed manner
- Enable self-service infrastructure provisioning by integrating an existing ServiceNow interface with automated workflows that generate pull requests for new Databricks resources, ensuring standardized and governed deployments
- Troubleshoot and resolve platform-related issues to ensure high availability and uninterrupted service, including proactive monitoring and management of Azure quota limits, cluster performance, resource utilization, and platform constraints
- Drive FinOps integration by onboarding Databricks service cost data into the existing FinOps (Umbrella) system, including configuring required data formats, enabling cost monitoring and reporting, and ensuring transparency and optimized resource usage across Azure and Databricks
- Stay up to date with industry best practices and emerging technologies, especially Terraform (Azure Provider, Databricks Provider)