Platform Engineer (Databricks) (7 – 10 Years)

Experience: 7 – 10 Years
Employment Type: Full-Time

Role Overview

We are looking for an experienced Platform Engineer with strong Databricks expertise to design, build, automate, and support enterprise-scale cloud data platforms. The ideal candidate will be responsible for enabling reliable, secure, and scalable Lakehouse platforms, driving platform automation, governance, observability, and operational excellence across cloud environments. You will collaborate with Data Engineers, Architects, DevOps teams, Security teams, and business stakeholders to deliver high-performing data and analytics platforms

Key Responsibilities

Platform Architecture & Engineering

- Design, implement, and maintain scalable, secure, and highly available Databricks-based Lakehouse platforms across Azure, AWS, or GCP environments.
- Establish platform engineering standards, cloud architecture patterns, governance frameworks, and operational best practices.
- Develop reusable platform templates and automation capabilities for enterprise-wide adoption.
- Support multi-environment strategies including Development, QA, UAT, and Production deployments.

Databricks Administration & Optimization

- Deploy, configure, and manage Databricks Workspaces, Clusters, SQL Warehouses, Unity Catalog, and Delta Lake environments.
- Implement cluster policies, workspace governance, access controls, and resource optimization strategies.
- Monitor and optimize Databricks workloads, job performance, cluster utilization, and platform costs.
- Enable enterprise-grade Lakehouse capabilities using Delta Lake and Databricks SQL.

Cloud Infrastructure & Automation

- Build and manage cloud infrastructure using Infrastructure-as-Code tools such as Terraform, ARM Templates, or CloudFormation.
- Manage cloud-native storage and security services including Azure Data Lake Storage (ADLS), AWS S3, Key Vault, IAM, Networking, and Secrets Management.
- Automate infrastructure provisioning, environment setup, and operational activities.
- Support cloud platform modernization and migration initiatives.
- Strong knowledge on Databricks administration and components.

CI/CD & DevOps

- Develop and maintain CI/CD pipelines for Databricks assets, infrastructure components, and platform services
- Implement release automation using Azure DevOps, GitHub Actions, Jenkins, GitLab CI/CD, or similar tools.
- Automate deployment of notebooks, workflows, libraries, and platform configurations.
- Ensure continuous integration, testing, deployment, and monitoring of platform components.

Security, Governance & Compliance





- Implement and manage role-based access control (RBAC), data security policies, encryption standards, and audit controls.
- Support enterprise governance using Unity Catalog, metadata management, and data lineage capabilities.
- Ensure compliance with organizational security, risk, and regulatory requirements.
- Collaborate with security teams to identify and remediate platform vulnerabilities.

Monitoring & Reliability Engineering

- Implement monitoring, logging, observability, alerting, and operational dashboards for platform services.
- Troubleshoot platform issues, bottlenecks, failures, and availability concerns.
- Drive platform reliability, scalability, resiliency, and disaster recovery capabilities.
- Create operational runbooks and platform documentation.

Collaboration & Innovation

- Partner with Data Engineers, Data Scientists, Architects, and Product Teams to enable data-driven solutions.
- Evaluate emerging cloud, analytics, and platform technologies.
- Develop Proof of Concepts (POCs) for new tools and capabilities.
- Maintain technical documentation, architecture diagrams, and best practices.

Required Skills & Qualifications

- 7–10 years of experience in Platform Engineering, Cloud Engineering, Data Platform Engineering, or related roles.
- Strong hands-on experience with Databricks platform administration, deployment, and optimization.
- Experience implementing Lakehouse architectures using Delta Lake and Databricks technologies.
- Strong knowledge of Databricks Workflows, Unity Catalog, Databricks SQL, Delta Live Tables (DLT), and Job Orchestration.
- Robust experience with any cloud platform (Azure, AWS, or GCP).
- Hands-on experience with Infrastructure as Code using Terraform, ARM Templates, or CloudFormation.
- Knowledge Python, PySpark, and Shell scripting experience.
- Experience with CI/CD tools such as Azure DevOps, GitHub Actions, Jenkins, or GitLab CI/CD.
- Experience implementing monitoring, logging, and observability frameworks.
- Strong understanding of networking, security, identity management, and access controls.
- Experience with cloud storage platforms such as ADLS, S3, and enterprise data platforms.
- Excellent troubleshooting, problem-solving,



and platform optimization skills.

Nice to Have

- Databricks Administration Certifications.
- Azure Administrator, Azure Solutions Architect, AWS Solutions Architect, or equivalent cloud certifications.
- Experience with Kubernetes, Docker, and container orchestration platforms.
- Knowledge of Snowflake, Synapse Analytics, or other modern analytics platforms.

Education

- Bachelor's degree in Computer Science, Information Technology, Engineering, or related field.

Case Study Example

Build and Operate an Enterprise Databricks Lakehouse Platform for Data Engineering, Analytics, and AI Workloads

Background:
A large enterprise requires a centralized cloud-native analytics platform to support multiple business domains. Existing environments had fragmented infrastructure, inconsistent governance, manual deployments, and operational challenges that impacted scalability, security, and time-to-market.

Challenge:

- Create a secure and scalable Databricks platform supporting multiple business units and workloads.
- Standardize platform deployment, governance, security, and operational processes.
- Enable self-service analytics while maintaining enterprise compliance.
- Automate infrastructure provisioning, CI/CD, monitoring, and access management.

Solution:

- Designed and implemented a cloud-native Databricks Lakehouse platform on Azure/AWS.
- Automated workspace and infrastructure deployment using Terraform and CI/CD pipelines.
- Implemented Unity Catalog for centralized governance, data access management, and lineage.
- Configured enterprise monitoring, logging, alerting, and platform observability.
- Established security controls including RBAC, encryption, network isolation, and secret management.
- Enabled scalable compute environments with optimized cluster policies and cost controls.
- Standardized deployment frameworks for data engineering, analytics, and ML workloads.
- Collaborated with enterprise architecture, security, and data engineering teams to drive platform adoption.

Impact:

- Reduced platform provisioning timelines from weeks to hours through automation.
- Improved operational efficiency and reliability through monitoring and self-healing capabilities.
- Enhanced governance, security, and compliance across analytics workloads.
- Enabled scalable Lakehouse architecture supporting enterprise data, analytics, and AI initiatives.
- Reduced operational costs through optimized cluster governance and resource utilization.
- Accelerated delivery of data products and analytics solutions across business functions.

📌 Platform Engineer (Telangana)
🏢 Crisil
📍 Telangana