Bangalore, Karnataka

Job Summary

This role requires a solid blend of software engineering, data engineering, cloud architecture, and performance optimization skills.

Key Responsibilities

Python Application Modernization

Refactor and convert object-oriented Python code into scalable PySpark-based distributed processing solutions.

Transform procedural and class-based business logic into DataFrame-based processing patterns.

Modernize legacy ETL frameworks into cloud-native Spark architectures.

Azure Databricks Development

Design, develop, and deploy enterprise-scale data pipelines using Azure Databricks.

Build reusable PySpark frameworks, libraries, and utility modules.

Develop Spark SQL transformations for large-scale analytics workloads.

Implement Delta Lake-based solutions using Bronze-Silver-Gold architecture.

Optimize Spark jobs for performance, scalability, and cost efficiency.

Data Engineering

Design robust ETL/ELT pipelines using:
Azure Databricks

Azure Data Factory

ADLS Gen2

Delta Lake

Azure Synapse Analytics

Process structured, semi-structured,



and unstructured data from multiple enterprise sources.

Implement data quality, reconciliation, validation, and monitoring frameworks.

Performance Engineering

Optimize Spark jobs using:
Partitioning

Bucketing

Caching

Broadcast Joins

Adaptive Query Execution

Delta Optimization Techniques

Reduce execution time and improve scalability for large-volume data workloads.

Skill Requirements

Azure Stack

Azure Databricks

Azure Data Factory (ADF)

Azure Data Lake Storage Gen2 (ADLS)

Programming

Python (Expert Level)

Object-Oriented Programming (OOP)

Advanced Python Design Patterns

PySpark

Spark SQL

SQL

Big Data Technologies

Apache Spark

Delta Lake

Data Lakehouse Architecture

Distributed Computing Concepts

Other Requirements

1.Relevant certifications in Azure Data Factory, Azure Databricks, SQL, Oracle, or Python would be a plus.
body.unify div.unify-button-container.unify-apply-now:

📌 Azure Senior Data Lead (Bengaluru)
🏢 HCLTech
📍 Bengaluru