Profile : Senior Azure Databricks Support Engineer

Location : Noida

Experience : 7+ Years

Contact : [email protected]

We are seeking a Senior Azure Databricks Support Engineer with robust expertise in production support, Azure cloud, Databricks, PySpark, Python, SQL, and ETL/ELT pipelines. The ideal candidate will be responsible for managing mission-critical production environments, resolving complex incidents, supporting deployments, optimizing platform performance, and collaborating with development teams to ensure seamless transition from support to production. This role combines production operations (60%) with development and automation (40%) to improve platform reliability, scalability, and operational efficiency.

Key Responsibilities

Production Support

- Provide L3 production support for Azure Databricks workloads, data pipelines, notebooks, workflows, and clusters.
- Monitor production environments and proactively identify performance bottlenecks.
- Troubleshoot failures in ETL/ELT pipelines and Spark jobs with minimal business impact.
- Handle critical production incidents, perform Root Cause Analysis (RCA), and implement permanent fixes.
- Ensure compliance with SLA, KPI, and ITIL support processes.
- Participate in 24x7 on-call and rotational production support.

Databricks Administration

- Manage Azure Databricks workspaces, clusters, jobs, libraries, and Unity Catalog.
- Configure autoscaling, cluster policies, security, and access controls.
- Optimize Spark jobs for performance, cost, and resource utilization.
- Monitor cluster health and recommend infrastructure improvements.

Development & Production Readiness

- Develop and enhance PySpark, Python, and SQL code for data processing.
- Support migration of code from Development QA UAT Production.
- Review deployment packages and validate production readiness.
- Assist developers in resolving production defects and performance issues.




- Perform code optimization and support release management activities.

Azure Cloud Operations

- Support Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), Azure Key Vault, Azure Monitor, and Log Analytics.
- Configure monitoring dashboards, alerts, and automated notifications.
- Work with infrastructure, networking, and security teams to maintain platform stability.
- Manage environment upgrades, patches, and capacity planning.

CI/CD & Automation

- Support Azure DevOps pipelines for automated deployments.
- Automate operational tasks using Python, PowerShell, or Shell scripting.
- Improve deployment processes through Infrastructure as Code (IaC) and automation.
- Maintain version control using Git.

Performance & Optimization

- Optimize Spark execution plans, partitioning, caching, and cluster utilization.
- Reduce job execution time and cloud infrastructure costs.
- Analyze job failures and recommend best practices for performance tuning.

Collaboration

- Work closely with Data Engineers, Data Architects, DevOps Engineers, Business Analysts, and Application Support teams.
- Create SOPs, knowledge base articles, runbooks, and operational documentation.
- Mentor junior support engineers and provide technical guidance.
- Participate in production release planning and post-deployment validation.

Required Skills

- Azure Databricks
- Apache Spark
- PySpark
- Python
- SQL
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS Gen2)
- Azure Synapse Analytics
- Azure DevOps
- Git
- CI/CD Pipelines
- Unity Catalog
- Delta Lake

Required Qualifications

- Bachelor's degree in Computer Science, Information Technology, or related field.
- 610 years of experience in Azure Data Engineering or Azure Databricks Production Support.
- Hands-on experience supporting enterprise-scale production environments.
- Strong knowledge of Spark architecture and Databricks optimization techniques.
- Experience in incident management, change management, and release management.
- Excellent troubleshooting and analytical skills.

📌 Senior Databricks Support Engineer (Noida)
🏢 Multiverse Solutions
📍 Noida