Databricks Platform Expert Key Responsibilities Databricks Platform Expert- Manage, configure, and administer Databricks workspaces, Clusters, SQL Warehouses, Serverless, jobs, and workspace objects.- Implement and manage Unity Catalog, including catalogs, schemas, tables, access controls, and data lineage.- Optimize cluster policies, auto-scaling strategies, and cost management for Serverless and Classic compute.- Serve as the SME for Databricks infrastructure, governance, and security best practices.- Monitor workspace performance, cluster stability, logs, job reliability, and platform health.- Implement CI/CD pipelines for notebooks, jobs, and Delta Live Tables using Git integration.- Support user provisioning, access controls (ACLs), secrets management, and workspace SSO.- Write productive Spark (PySpark / SQL / Scala) code for ETL, data transformations, and pipeline optimizations.- Assist data engineering teams with Spark job debugging, performance tuning, and code reviews.- Build and maintain production-grade pipelines leveraging Delta Lake, Databricks Jobs, and DLT.- Implement and manage RBAC, SCIM provisioning, AIM, service principals, and cluster access controls.- Ensure compliance with enterprise data governance, audit, and logging requirements.- Manage secrets Key Vault and enforce secure credential handling.- Support audit reports, compliance reviews,



and workspace security configuration.- Monitor job failures, cluster lifecycle performance, and system events using Databricks logs and cloud-native monitoring tools (Azure Monitor).- Create automated alerts and observability dashboards for platform usage, cost, and performance.- Troubleshoot Databricks runtime issues, library conflicts, and Spark execution failures.- Collaborate with cloud and network teams on VNet, peering, and private-link connectivity issues.- Develop cost governance policies for cluster sizes, job policies, and SQL Warehouse tiers.- Identify opportunities to reduce cost via autoscaling, spot instances (classic clusters), and job consolidation. Required Qualifications- 4-6 years of experience working with Databricks as an administrator or data engineer.- Strong expertise in Apache Spark programming (PySpark preferred; SQL or Scala is a plus).- Hands-on experience with Databricks Jobs, cluster configuration, SQL Warehouses, and Unity Catalog.- Deep understanding of Delta Lake, ACID transactions, and lakehouse architecture.- Experience with Git, CI/CD, and DevOps concepts for data engineering workflows.- Knowledge of cloud platforms ( Azure).- Familiarity with IAM, networking basics, monitoring tools, and security patterns

📌 Databricks Platform Expert (Pune)
🏢 Michelin
📍 Pune