Azure Senior Data Lead

Experience: Not Available to Not Available years

Location: Bangalore, India

Skills: Databricks, Apache Spark, Delta Lake, Databricks REST API, SDK, Python, SQL, PySpark, Spark SQL, AWS, Azure, GCP, Docker, Kubernetes, Terraform, CI/CD, MLflow, Mosaic AI, Prometheus, Grafana, OpenTelemetry, Datadog

Job Summary
Design and build optimization capabilities for Databricks - spanning Spark tuning, cluster right-sizing, job orchestration, DBU consumption, and Delta Lake storage.
• Translate platform expertise into product features - detection rules, recommendation engines, and protected automated actions for production environments.
• Build POCs to validate optimization ideas, demonstrate value, and support pre-sales engagements.
• Partner cross-functionally with backend, AI/ML, and data engineering teams to ship features end-to-end

Key Responsibilities
Design and build optimization capabilities for Databricks - spanning Spark tuning, cluster right-sizing, job orchestration, DBU consumption, and Delta Lake storage.
• Translate platform expertise into product features - detection rules, recommendation engines, and safe automated actions for production environments.
• Build POCs to validate optimization ideas, demonstrate value, and support pre-sales engagements.
• Partner cross-functionally with backend, AI/ML, and data engineering teams to ship features end-to-end

Skill Requirements
Engineering experience; hands-on exp in Databricks in production.
• Apache Spark internals - Catalyst optimizer, Tungsten engine, AQE, DAG scheduler, shuffle behavior, partitioning, broadcast/sort-merge joins, data skew handling, and Spark 4.0 capabilities.
• Databricks platform depth - Delta Lake (transaction log,



OPTIMIZE, ZORDER, vacuum, liquid clustering, schema evolution, time travel, CDC/merge), Lakeflow Declarative Pipelines, Unity Catalog (governance, lineage, fine-grained access), Photon engine, Databricks Workflows, Lakebase, and all cluster types (job, all-purpose, serverless SQL, serverless compute).
• Databricks REST API & SDK - programmatic management of clusters, jobs, permissions, and workspace configuration.
• Performance tuning - Spark UI interpretation, physical plans, shuffle/skew/spill diagnosis, join optimization, caching strategies, and Photon adoption decisions.
• Cost optimization - DBU forecasting, cluster sizing, autoscaling policies, spot vs. on-demand trade-offs, instance pools, job-vs-all-purpose decisions, predictive optimization, serverless economics (Performance vs. Standard mode, serverless GPU, egress, DBU trade-offs).
• Advanced Python & expert SQL; deep PySpark and Spark SQL internals.
• Cloud platforms (AWS/Azure/GCP) - IAM, networking, storage (S3/ADLS/GCS), and cloud-native services underpinning Databricks.
• Experience with Docker, Kubernetes, Terraform, and modern CI/CD pipelines.
• Strong fundamentals in data structures, algorithms, distributed systems, and large-scale system design.
• MLflow, Mosaic AI ecosystem (Agent Framework, Agent Bricks, AI Gateway, Vector Search), feature stores, Databricks SQL Warehouses, or Databricks Asset Bundles.
• FinOps practices and cost-attribution models for data platforms.
• Observability tools - Prometheus, Grafana, OpenTelemetry, Datadog.
• Contributions to open-source Spark/Delta/Databricks projects

Other Requirements
Databricks certifications a plus
• BS/MS in Computer Science, Engineering, or related field

📌 Azure Senior Data Lead (Bengaluru)
🏢 HCL Technologies
📍 Bengaluru