Bangalore, Karnataka

Job Summary

We are seeking an experienced Senior Azure Databricks Data Engineer to lead the modernization and migration of existing Python Object-Oriented (OOP) applications into scalable PySpark and Spark SQL-based data processing solutions on Azure Databricks.

The ideal candidate will possess deep expertise in Python OOP, software engineering principles, distributed computing, PySpark, Spark SQL, and Azure Databricks . The primary responsibility will be to analyze existing Python applications, redesign processing logic for distributed execution, optimize performance, and build enterprise-grade data engineering solutions on Azure.

This role requires a robust blend of software engineering, data engineering, cloud architecture, and performance optimization skills .

Key Responsibilities

Python Application Modernization

Analyze existing Python OOP applications, libraries, and frameworks.

Refactor and convert object-oriented Python code into scalable PySpark-based distributed processing solutions.





Identify bottlenecks in single-node Python applications and redesign them for Spark execution.

Transform procedural and class-based business logic into DataFrame-based processing patterns.

Modernize legacy ETL frameworks into cloud-native Spark architectures.

Azure Databricks Development

Design, develop, and deploy enterprise-scale data pipelines using Azure Databricks.

Build reusable PySpark frameworks, libraries, and utility modules.

Develop Spark SQL transformations for large-scale analytics workloads.

Implement Delta Lake-based solutions using Bronze-Silver-Gold architecture.

Optimize Spark jobs for performance, scalability, and cost efficiency.

Data Engineering

Design robust ETL/ELT pipelines using:
Azure Databricks

Azure Data Factory

ADLS Gen2

Delta Lake

Azure Synapse Analytics

Process structured, semi-structured, and unstructured data from multiple enterprise sources.

Implement data quality, reconciliation, validation, and m

📌 Azure Senior Data Lead (Bengaluru)
🏢 HCLTech
📍 Bengaluru