Azure Senior Data Lead

Experience: Not Available to Not Available years

Location: Gautam Buddha Nagar, India

Skills: Databricks, Apache Spark, Databricks clusters, Databricks jobs, Databricks workflows, Databricks notebooks, Workspace security, IAM, Secrets management, SQL, Python, Scala, ADLS Gen2, S3, Terraform, CI/CD, Azure DevOps, GitHub, GitLab, Databricks REST APIs, Incident management, Root cause analysis, ITIL

Job Summary
Job Description Databricks Administrator – Platform Support Role Summary We are looking for an experienced Databricks Administrator to provide platform support, administration, and operational stability for Databricks environments. The role focuses on workspace management, security, performance, cost optimization, monitoring, and incident support, ensuring a reliable and scalable analytics platform for business and engineering teams.

Key Responsibilities
Platform Administration & Support
• Administer and maintain Databricks workspaces across Development, Test, and Production environments
• Manage cluster configurations, policies, pools, and job scheduling
• Provide L2/L3 platform support for Databricks-related incidents, service requests, and problem management
• Perform health checks and ensure high availability and platform stability
• Support workspace onboarding/offboarding for users and teams
Security & Access Management
• Configure and manage user access, groups, and permissions using Databricks IAM and cloud IAM tools
• Implement role-based access control (RBAC) and workspace security best practices




• Manage secrets using Databricks Secrets integrated with cloud key vaults
• Support compliance requirements (SOX, GDPR, ISO, etc.)
Cloud & Infrastructure Integration
• Work with cloud services such as Azure (ADLS, ADF, Key Vault) or AWS (S3, IAM, Glue)
• Manage networking setups including VNET/VPC, private endpoints, firewall rules
• Coordinate with cloud platform teams for infrastructure-related issues
Monitoring, Performance & Cost Optimization
• Monitor cluster utilization, job performance, and platform metrics
• Identify and resolve performance bottlenecks across Spark workloads
• Implement cost optimization strategies (cluster sizing, auto-scaling, auto-termination)
• Generate usage and cost reports for stakeholders
Automation & DevOps Support
• CI/CD pipelines for Databricks artifacts using tools like Azure DevOps, GitHub, or GitLab
• Automate workspace and cluster management using Databricks REST APIs, Terraform, or scripting
• Maintain configuration consistency across environments
Documentation & Governance
• Create and maintain platform runbooks, SOPs, and support documentation
• Enforce platform governance standards and coding best practices
• Provide usage guidelines and operational best practices to engineering teams




Required Skills & Qualifications
Technical Skills
• Robust hands-on experience administering Databricks on Azure or AWS
• Solid understanding of Apache Spark architecture
• Experience with: Databricks clusters, jobs, workflows, and notebooks
• Workspace security, IAM, and secrets management
• Monitoring and troubleshooting Spark jobs
• Proficiency in SQL and basic Python / Scala for operational support
• Experience with cloud storage and services (ADLS Gen2 / S3)
• Familiarity with Terraform or Infrastructure as Code is a strong plus
Platform & Support Skills
• Experience working in production support environments
• Strong incident management and root cause analysis skills
• Ability to handle multiple priorities in a 24x7 or on-call support model
• Good understanding of ITIL processes (Incident, Problem, Change Management)
Preferred Qualifications
• Databricks Certified: Databricks Data Engineer Associate / Professional
• Databricks Platform Administrator (preferred)
• Experience supporting enterprise-scale data platforms
• Knowledge of Delta Lake, Unity Catalog, and data governance models
• Exposure to DevOps, CI/CD pipelines, and automation tools
Soft Skills
• Strong analytical and problem-solving skills
• Excellent communication and stakeholder management abilities
• Proactive mindset with attention to detail
• Ability to work collaboratively across data, cloud, and security teams
Skill Requirements
null
Other Requirements
null

📌 Azure Senior Data Lead (Noida)
🏢 HCL Technologies
📍 Noida