Role & responsibilities

Job Description Databricks Production Support Engineer (L2/L3 Production Support)

Job Title - Databricks Support Engineer

Experience - 4 to 8 Years

Location - Hybrid Noida

Employment Type - Full-Time

About the Role

We are looking for a highly motivated Databricks Support Engineer to join our Data Engineering team. The ideal candidate will be responsible for supporting and maintaining enterprise-scale data platforms built on the Databricks Lakehouse Platform. This role requires hands-on experience in production support, Databricks administration, Apache Spark, Delta Lake, Airflow, SQL, and cloud technologies, with a strong focus on troubleshooting, performance optimization, and ensuring high availability of data pipelines.

The successful candidate should be comfortable working in a quick-paced production environment, collaborating with cross-functional teams, and resolving complex technical issues while meeting strict SLAs.

Key Responsibilities

- Provide L2/L3 production support for Databricks-based data platforms and analytics solutions.
- Monitor, troubleshoot, and resolve issues related to Databricks Jobs, Workflows, Spark applications, and ETL pipelines.
- Investigate production incidents, perform Root Cause Analysis (RCA), and implement preventive solutions.
- Support ETL/ELT pipelines built using PySpark, Spark SQL, and Delta Lake.
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Manage Delta Lake operations including MERGE, OPTIMIZE, VACUUM, schema evolution, and incremental data loads.
- Troubleshoot and maintain Apache Airflow DAGs, scheduling, retries, and dependency management.
- Support CI/CD deployments using Azure DevOps, GitHub Actions, or Jenkins.
- Work with cloud storage services, REST APIs, and enterprise data sources.
- Monitor cluster utilization, autoscaling, job execution, and resource consumption.
- Collaborate with Data Engineering and DevOps teams to improve platform reliability and operational excellence.
- Ensure compliance with security, governance, and operational best practices using Unity Catalog.
- Maintain technical documentation, runbooks, and production support procedures.

Required Technical Skills

Databricks

- Databricks Lakehouse Platform
- Databricks Workspaces
- Databricks Jobs & Workflows
- Delta Lake
- Unity Catalog
- Databricks SQL
- Cluster Management
- Performance Tuning
- Job Monitoring & Troubleshooting

Data Engineering

- Apache Spark
- PySpark
- Spark SQL
- SQL
- Delta MERGE Operations
- Incremental Data Loading




- Change Data Capture (CDC)
- Data Partitioning
- Z-Ordering
- OPTIMIZE & VACUUM
- Adaptive Query Execution (AQE)
- Medallion Architecture (Bronze, Silver, Gold)

Workflow Orchestration

- Apache Airflow
- DAG Scheduling
- Retry Mechanisms
- Dependency Management
- Pipeline Monitoring

Cloud Platforms

Experience with one or more cloud platforms:

- Microsoft Azure (Preferred)
- Amazon Web Services (AWS)
- Google Cloud Platform (GCP)

CI/CD & DevOps

- Git
- Azure DevOps
- GitHub Actions
- Jenkins
- Deployment Automation
- Infrastructure as Code concepts

Databases

Experience with one or more:

- Databricks SQL
- Snowflake
- SQL Server
- PostgreSQL
- Amazon Redshift
- Google BigQuery

Preferred Qualifications

- Experience supporting enterprise-scale production data platforms.
- Strong understanding of Spark optimization and distributed computing.
- Experience with monitoring and alerting tools.
- Hands-on experience with Unity Catalog and data governance.
- Knowledge of cloud security and access management.

Good to Have

- Terraform
- Kubernetes
- Kafka
- dbt
- MLflow
- REST API Integrations

Technical Interview Focus Areas

Candidates should have practical experience and be able to explain:

Project Experience

- Explain your end-to-end project architecture.
- Describe your role and responsibilities.
- Explain the complete data flow from source to reporting.
- Discuss production issues you have resolved.
- Explain the largest volume of data you have processed.

Databricks & Lakehouse

- Databricks Architecture
- Lakehouse Architecture
- Delta Lake
- ACID Transactions
- Databricks Jobs & Workflows
- Unity Catalog
- Databricks SQL
- Difference between Databricks, Snowflake, Redshift, and BigQuery

Unity Catalog & Data Governance

- What is Unity Catalog?
- Why is Unity Catalog required?
- Three-level namespace (Catalog Schema Table)
- Access Control & Permissions
- Data Lineage
- Governance and Security

Schema Design

Candidates should be able to explain:

- Which schema was used in their project.
- Why that schema was selected.
- Difference between Star Schema and Snowflake Schema.




- Schema evolution and handling source changes.
- Partitioning strategy and table design.

Apache Spark

- Lazy Evaluation
- Narrow vs. Wide Transformations
- Shuffle Optimization
- Broadcast Joins
- Caching vs. Persist
- Adaptive Query Execution (AQE)
- Repartition vs. Coalesce

Delta Lake

- Delta MERGE
- Incremental Loading
- Time Travel
- Schema Evolution
- OPTIMIZE
- VACUUM
- Z-Ordering
- File Compaction

Apache Airflow

- DAG Design
- Scheduling
- Retry Logic
- Dependency Management
- Triggering Databricks Jobs
- Production Troubleshooting

Performance & Cost Optimization

- Cluster Sizing
- Autoscaling
- Job Clusters vs. All-Purpose Clusters
- Compute Cost Optimization
- Spark Performance Tuning

Scenario-Based Problem Solving

Candidates should be able to solve real-world scenarios such as:

- Optimizing a Delta MERGE for large incremental datasets within a strict SLA.
- Troubleshooting failed Databricks Jobs or Airflow DAGs.
- Resolving duplicate data issues in Delta tables.
- Reducing Spark job execution time through tuning and optimization.
- Designing scalable data pipelines for high-volume enterprise workloads.
- Performing Root Cause Analysis (RCA) for production incidents.

Education

- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.

Preferred candidate profile

We are looking for a Databricks Support Engineer with 48 years of experience in supporting enterprise data platforms built on Databricks. The ideal candidate should have hands-on expertise in Databricks, Apache Spark (PySpark), Delta Lake, Apache Airflow, SQL, and Azure/AWS/GCP, with strong experience in L2/L3 production support, troubleshooting, performance tuning, and ETL/ELT pipeline management. The role involves monitoring Databricks Jobs and Workflows, resolving production incidents, performing root cause analysis (RCA), optimizing Spark jobs, managing Delta Lake operations (MERGE, OPTIMIZE, VACUUM), and ensuring SLA compliance. Experience with Unity Catalog, Medallion Architecture, CI/CD (Azure DevOps/GitHub/Jenkins), and cloud-based data engineering is highly preferred.

NOTE - THIS POSITION IS ONLY FOR NOIDA LOCATION AND NO PROFILES WILL BE CONSIDERED OUT OF DELHI NCR

Contact -

Name - Sachin Sharma - (phone hidden)

Email - [email protected]

Name - Siddharth Mathur

Call / What's App Resume - (phone hidden)

Email - [email protected]

Manager - Global Talent Acquisition

MOPTRA INFOTCH PVT LTD

📌 Data Bricks Production Support (Noida)
🏢 Moptra Infotech
📍 Noida