Bangalore, Karnataka
Job Summary

We are seeking an experienced Senior Azure Databricks Data Engineer to lead the modernization and migration of existing Python Object-Oriented (OOP) applications into scalable PySpark and Spark SQL-based data processing solutions on Azure Databricks.

The ideal candidate will possess deep expertise in Python OOP, software engineering principles, distributed computing, PySpark, Spark SQL, and Azure Databricks . The primary responsibility will be to analyze existing Python applications, redesign processing logic for distributed execution, optimize performance, and build enterprise-grade data engineering solutions on Azure.

This role requires a robust blend of software engineering, data engineering, cloud architecture, and performance optimization skills .

Key Responsibilities
Python Application Modernization
Analyze existing Python OOP applications, libraries, and frameworks.
Refactor and convert object-oriented Python code into scalable PySpark-based distributed processing solutions.
Identify bottlenecks in single-node Python applications and redesign them for Spark execution.
Transform procedural and class-based business logic into DataFrame-based processing patterns.
Modernize legacy ETL frameworks into cloud-native Spark architectures.
Azure Databricks Development
Design, develop, and deploy enterprise-scale data pipelines using Azure Databricks.
Build reusable PySpark frameworks, libraries, and utility modules.
Develop Spark SQL transformations for large-scale analytics workloads.




Implement Delta Lake-based solutions using Bronze-Silver-Gold architecture.
Optimize Spark jobs for performance, scalability, and cost efficiency.
Data Engineering

Design robust ETL/ELT pipelines using:
Azure Databricks
Azure Data Factory
ADLS Gen2
Delta Lake
Azure Synapse Analytics

Process structured, semi-structured, and unstructured data from multiple enterprise sources.

Implement data quality, reconciliation, validation, and monitoring frameworks.

Performance Engineering

Optimize Spark jobs using:
Partitioning
Bucketing
Caching
Broadcast Joins
Adaptive Query Execution
Delta Optimization Techniques

Benchmark converted PySpark applications against original Python implementations.

Reduce execution time and improve scalability for large-volume data workloads.

Skill Requirements
Azure Stack
Azure Databricks
Azure Data Factory (ADF)
Azure Data Lake Storage Gen2 (ADLS)
Programming
Python (Expert Level)
Object-Oriented Programming (OOP)
Advanced Python Design Patterns
PySpark
Spark SQL
SQL
Big Data Technologies
Apache Spark
Delta Lake
Data Lakehouse Architecture
Distributed Computing Concepts
Other Requirements
1.Relevant certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python would be a plus.
#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-

📌 Azure Senior Data Lead (India)
🏢 HCLTech
📍 India