Job Summary

Data Platform Engineer with a robust focus on Databricks and Azure to design, build, and operate a scalable and secure data platform. You will work at the intersection of infrastructure, data engineering, and cloud automation to establish an enterprise platform for analytics and AI workloads.

LONGDESCRIPTION section. 2 of 6.Section Title: Key Responsibilities

Key Responsibilities
Automate the provisioning of Databricks and Azure platform resources using Terraform, adhering to Infrastructure-as-Code (IaC) best practices
Design and develop reusable, modular Terraform components to enable scalable and maintainable deployments within the Databricks platform
Implement and maintain CI/CD pipelines using Azure DevOps for automated infrastructure and platform deployments
Provision new infrastructure using customized Databricks Terraform modules to support requests for current catalogs, job policies, and clusters in a standardized, automated, and governed manner




Enable self-service infrastructure provisioning by integrating an existing ServiceNow interface with automated workflows that generate pull requests for recent Databricks resources, ensuring standardized and governed deployments
Troubleshoot and resolve platform-related issues to ensure high availability and uninterrupted service, including proactive monitoring and management of Azure quota limits, cluster performance, resource utilization, and platform constraints
Drive FinOps integration by onboarding Databricks service cost data into the existing FinOps (Umbrella) system, including configuring required data formats, enabling cost monitoring and reporting, and ensuring transparency and optimized resource usage across Azure and Databricks
Stay up to date with industry best practices and emerging technologies, especially Terraform (Azure Provider, Databricks Provider)

📌 Azure Devops + Databricks + Terraform Pune Location 5days Wfo (India)
🏢 HCLTech
📍 India