Databricks Platform Expert
- - - - - - - - - - - -
MISSION : We are seeking a highly skilled Databricks expert with hands-on experience in Apache Spark programming and solid knowledge of cloud-based data engineering platforms. The ideal candidate will be responsible for managing, optimizing, and supporting our Databricks environment, ensuring high availability, security, performance, and efficient usage across teams. This role combines platform administration with practical Spark development to support data engineering and analytics workloads.
Key Responsibilities
Databricks Platform Expert
- Manage, configure, and administer Databricks workspaces, Clusters, SQL Warehouses, Serverless, jobs, and workspace objects.
- Implement and manage Unity Catalog, including catalogs, schemas, tables, access controls, and data lineage.
- Optimize cluster policies, auto-scaling strategies, and cost management for Serverless and Classic compute.
- Serve as the SME for Databricks infrastructure, governance, and security best practices.
- Monitor workspace performance, cluster stability, logs,
job reliability, and platform health.
- Implement CI/CD pipelines for notebooks, jobs, and Delta Live Tables using Git integration.
- Support user provisioning, access controls (ACLs), secrets management, and workspace SSO.
- Write efficient Spark (PySpark / SQL / Scala) code for ETL, data transformations, and pipeline optimizations.
- Assist data engineering teams with Spark job debugging, performance tuning, and code reviews.
- Build and maintain production-grade pipelines leveraging Delta Lake, Databricks Jobs, and DLT.
- Implement and manage RBAC, SCIM provisioning, AIM, service principals, and cluster access controls.
- Ensure compliance with enterprise data governance, audit, and logging requirements.
- Manage secrets Key Vault and enforce secure credential handling.
- Support audit reports, compliance reviews, and workspace security configuration.
- Monitor job failures, cluster life
📌 Databricks Platform Expert (Pune)
🏢 Michelin
📍 Pune