(Note: Candidates below 6 years of IT experience shall not be considered)
Job Locations :
Anywhere in India
JOB DESCRIPTION
Must-Have Skills:
- Strong hands-on experience with
Azure Databricks
- Expertise in
Apache Spark (PySpark)
and
Spark SQL
- Solid programming skills in
Python
and
SQL
- Experience with
Delta Lake
including ACID transactions and schema evolution
- Hands-on experience in
ETL/ELT pipeline development
- Strong understanding of
Lakehouse Architecture
- Experience building and maintaining
Medallion Architecture (Bronze, Silver, Gold)
data layers
- Experience with
batch and streaming data processing
- Experience with
Azure Data Factory (ADF)
and Azure Data Engineering ecosystem
- Strong understanding of
Data Modeling
and Analytics data structures
- Experience with
Unity Catalog
or Data Governance frameworks
- Experience in performance tuning, query optimization, partitioning, and caching
- Data Engineering experience
- Azure Databricks experience
Good-to-Have Skills:
- Git, Azure DevOps, GitHub Actions
- CI/CD implementation and deployment automation
- Terraform (Infrastructure as Code)
- Docker and Kubernetes
- Airflow
- Kafka or Azure Event Hub
- Power BI or Tableau
- Databricks Workflows and Job Orchestration
- Cluster Management and Optimization
- Experience with Delta Table features such as MERGE, UPSERT, and Time Travel
- BFSI domain experience
Key Responsibilities:
- Design, develop, and optimize scalable batch and streaming data pipelines using Azure Databricks.
- Develop data processing frameworks using PySpark and Spark SQL.
- Implement Delta Lake-based storage solutions to ensure high reliability and data consistency.
- Build and maintain Bronze, Silver, and Gold data layers within a Lakehouse architecture.
- Ingest data from APIs, databases, Azure storage, and streaming platforms.
- Develop scalable data models supporting analytics and reporting requireme