Role: Tech Lead – Databricks / Data Platform

Location: Remote

Employment Type: Full-Time

Experience: 10–12 Years

JOB SUMMARY

We are looking for an experienced Tech Lead – Databricks / Data Platform to lead the design, development, and optimization of a scalable, secure, and reliable cloud-based data platform. The role will provide technical leadership across Databricks, Azure, CI/CD, Infrastructure as Code, orchestration, data security, FinOps, and platform automation.

The ideal candidate will be hands-on, technically strong, and experienced in leading engineering teams while partnering closely with Data Engineers, Analytics Engineers, Architects, Security, and Cloud teams to deliver enterprise-grade data solutions.

KEY RESPONSIBILITIES

- Lead the architecture, design, and implementation of Databricks-based data platforms and cloud-native data pipelines.

- Design and implement robust CI/CD pipelines for data ingestion, transformation, dbt, SQL, notebooks, and Databricks workloads.

- Lead end-to-end orchestration of Databricks Jobs and Workflows, including ingestion, transformation, data quality checks, and dependencies.

- Define and implement automated testing and quality gates within CI/CD pipelines, including schema, data model, and contract validation.

- Drive Infrastructure as Code (IaC) practices using Terraform and reusable modules for Databricks and Azure infrastructure.

- Automate Databricks platform operations, including workspace and cluster provisioning, runtime and library management, job deployment, configuration, and environment management.

- Establish and maintain Identity and Access Management (IAM) across Databricks and Azure, including RBAC, TBAC, service principals, groups, roles, and workspace/cluster/table-level access controls.

- Work closely with the EDP Architect and Security teams to implement secure and scalable Unity Catalog and data governance patterns.





- Develop reusable Terraform modules, Databricks job templates, Airflow DAG patterns, and engineering frameworks to accelerate onboarding of new projects.

- Define and govern the code promotion and release management process across development, QA, and production environments.

- Lead end-to-end orchestration using managed Airflow, ensuring reliable scheduling, dependency management, monitoring, and recovery.

- Define and track platform SLAs, SLOs, and SLIs, and lead incident triage, root cause analysis, and corrective actions.

- Implement FinOps best practices for monitoring, optimizing, and allocating Databricks and cloud infrastructure costs.

- Continuously evaluate and improve platform tooling, architecture, reliability, security, performance, and developer productivity.

- Provide technical guidance and mentorship to engineers and establish engineering best practices, standards, and reusable patterns.

- Partner with Data Engineering, Analytics, Architecture, Security, and Infrastructure teams to ensure seamless delivery of enterprise data solutions.

- Contribute to cloud infrastructure, cybersecurity, disaster recovery, monitoring, logging, and operational readiness.

- Drive performance optimization and reliability improvements for production data workloads.

REQUIRED QUALIFICATIONS

- 10–12 years of experience in Data Engineering, Cloud Engineering, DevOps, Platform Engineering, SRE, or related technical disciplines.

- Strong hands-on experience with Databricks in production environments, including workspace and cluster management,



Jobs/Workflows, Unity Catalog, and integration with orchestration platforms.

- Strong experience with Microsoft Azure and cloud-native data platforms.

- Strong knowledge of CI/CD and Git-based development workflows, using tools such as Azure DevOps, GitHub Actions, GitLab CI, or similar.

- Strong experience with Terraform / Infrastructure as Code and cloud infrastructure automation.

- Hands-on experience with Apache Airflow or managed Airflow for data pipeline orchestration.

- Strong programming/scripting skills in Python, Bash, or PowerShell.

- Experience implementing automated testing, validation, and quality gates for data pipelines and data models.

- Experience with production workload management, including monitoring, logging, troubleshooting, performance tuning, and incident management.

- Strong understanding of cloud security, IAM, RBAC, data access controls, and governance.

- Experience with FinOps, cloud cost optimization, and resource utilization monitoring.

- Ability to provide technical leadership, mentor engineers, and drive engineering standards across teams.

- Strong communication and stakeholder management skills with the ability to work effectively with technical and business stakeholders.

PREFERRED QUALIFICATIONS

- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent skilled experience.

- Experience with dbt, SQL, Delta Lake, and modern data engineering practices.

- Experience with Unity Catalog and enterprise data governance.

- Experience designing reusable platform frameworks and accelerators.

- Experience in CPG, retail, manufacturing, or distribution environments.

- Experience with disaster recovery, business continuity, and highly available cloud architectures.

- Databricks or Azure certifications are a plus.

📌 Tech Lead – Databricks / Data Platform (Pune)
🏢 Palo Alto Labs
📍 Pune