06 To 10 Years (Mandatory)
(Note: Candidates below 6 years of IT experience shall not be considered)
Job Locations :
Anywhere in India
JOB DESCRIPTION
Must-Have Skills:
Strong hands-on experience with
Azure Databricks
Expertise in
Apache Spark (PySpark)
and
Spark SQL
Strong programming skills in
Python
and
SQL
Experience with
Delta Lake
including ACID transactions and schema evolution
Hands-on experience in
ETL/ELT pipeline development
Strong understanding of
Lakehouse Architecture
Experience building and maintaining
Medallion Architecture (Bronze, Silver, Gold)
data layers
Experience with
batch and streaming data processing
Experience with
Azure Data Factory (ADF)
and Azure Data Engineering ecosystem
Robust understanding of
Data Modeling
and Analytics data structures
Experience with
Unity Catalog
or Data Governance frameworks
Experience in performance tuning, query optimization, partitioning, and caching
Data Engineering experience
Azure Databricks experience
Good-to-Have Skills:
Git, Azure DevOps, GitHub Actions
CI/CD implementation and deployment automation
Terraform (Infrastructure as Code)
Docker and Kubernetes
Airflow
Kafka or Azure Event Hub
Power BI or Tableau
Databricks Workflows and Job Orchestration
Cluster Management and Optimization
Experience with Delta Table features such as MERGE, UPSERT, and Time Travel
BFSI domain experience
Key Responsibilities:
Design, develop, and optimize scalable batch and streaming data pipelines using Azure Databricks.
Develop data processing frameworks using PySpark and Spark SQL.
Implement Delta Lake-based storage solutions to ensure high reliability and data consistency.
Build and maintain Bronze, Silver, and Gold data layers within a Lakehouse architecture.
Ingest data from APIs, databases, Azure storage, and streaming platforms.
Develop scalable data models supporting analytics and reporting requirements.
Implement data quality, validation