Azure Databricks Data EngineerAbout the Role

We are looking for an experienced Azure Databricks Data Engineer to design, build, and maintain scalable data pipelines and modern cloud data solutions.

The ideal candidate has strong hands-on experience with Azure Databricks, PySpark, Python, SQL, Azure Data Factory, ADLS Gen2, and Delta Lake and is comfortable working with large datasets and production data pipelines.

You will work closely with software engineers, data scientists, analysts, and technical stakeholders to build reliable, scalable, and high-quality data solutions.

Key Responsibilities

- Design, develop, and maintain scalable ETL/ELT data pipelines using Azure Databricks, PySpark, Python, and SQL.
- Build data ingestion and transformation pipelines using Azure Data Factory and Azure Databricks.
- Work with Azure Data Lake Storage Gen2 (ADLS Gen2) and other Azure data services.
- Develop and maintain Delta Lake tables and lakehouse data architectures.
- Implement data processing using Apache Spark / PySpark.
- Design efficient data models for analytics, reporting, and downstream applications.
- Build and maintain Bronze, Silver, and Gold data layers where applicable.
- Implement data quality, validation, monitoring, and error-handling processes.
- Optimize Spark jobs, SQL queries, cluster configurations, partitioning, and data pipelines for performance and cost efficiency.
- Implement appropriate security and governance practices using Unity Catalog.
- Develop and maintain CI/CD workflows for Databricks notebooks, jobs, pipelines, and related infrastructure.
- Use Git and Azure DevOps for source control, collaboration, and deployment.
- Monitor production pipelines and troubleshoot failures, performance issues, and data-quality problems.




- Collaborate with data scientists, analysts, software engineers, architects, and business stakeholders.
- Document data pipelines, data models, technical processes, and operational procedures.

Required Qualifications
- 3+ years of professional experience in Data Engineering.
- Strong hands-on experience with Azure Databricks.
- Strong experience with Python and PySpark.
- Strong SQL skills, including complex queries, joins, CTEs, window functions, and performance optimization.
- Experience building production-grade ETL/ELT pipelines.
- Experience with Azure Data Factory (ADF).
- Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
- Robust understanding of Apache Spark and distributed data processing.
- Experience with Delta Lake and lakehouse architecture.
- Experience with Git-based development workflows.
- Understanding of CI/CD and deployment practices.
- Experience troubleshooting and monitoring production data pipelines.
- Understanding of data modeling, data quality, and data governance.

Preferred Qualifications
- Experience with Unity Catalog.
- Experience with Azure DevOps.
- Experience implementing CI/CD for Databricks.
- Experience with Azure Monitor.
- Experience with Microsoft Entra ID / Azure identity and access management.
- Experience with Azure Key Vault.
- Experience with Azure Synapse Analytics.




- Experience working with REST APIs and semi-structured data such as JSON.
- Experience with streaming or near-real-time data pipelines.
- Experience with Databricks Workflows / Jobs.
- Experience with infrastructure-as-code or cloud automation.
- Microsoft Azure or Databricks certifications are a plus.

Technical Environment

Cloud: Microsoft Azure

Data Platform: Azure Databricks

Processing: Apache Spark, PySpark

Languages: Python, SQL

Storage: ADLS Gen2, Delta Lake

Orchestration: Azure Data Factory, Databricks Jobs/Workflows

DevOps: Git, Azure DevOps, CI/CD

Governance: Unity Catalog

Monitoring: Azure Monitor / Databricks monitoring

What We're Looking For

We are looking for someone who is hands-on and can independently take a data engineering requirement from design through production.

You should be comfortable writing code, debugging pipelines, optimizing Spark workloads, working with cloud data platforms, and explaining technical decisions clearly.

Candidates should be able to demonstrate real-world experience building and operating production Azure Databricks data pipelines, not just theoretical knowledge.

Education

Bachelor's degree in Computer Science, Information Technology, Engineering, Data Science, or a related field is preferred.

Equivalent practical experience will also be considered.

Interview Process The interview process may include:

- Resume / technical screening
- Azure & Databricks technical interview
- PySpark / SQL assessment
- Practical data engineering or architecture discussion
- Final interview

Pay: ₹1,200,000.00 - ₹2,200,000.00 per year

Benefits

- Flexible schedule

Work Location: Hybrid remote in Hyderabad, Telangana 500081

📌 Azure Databricks Data Engineer (Hyderabad)
🏢 NovaFlow Ai
📍 Hyderabad