(Note: Candidates below 6 years of IT experience shall not be considered)
Job Locations : Anywhere in India
JOB DESCRIPTION
Must-Have Skills:
Strong hands-on experience with Azure Databricks
Expertise in Apache Spark (PySpark) and Spark SQL
Solid programming skills in Python and SQL
Experience with Delta Lake including ACID transactions and schema evolution
Hands-on experience in ETL/ELT pipeline development
Solid understanding of Lakehouse Architecture
Experience building and maintaining Medallion Architecture (Bronze, Silver, Gold) data layers
Experience with batch and streaming data processing
Experience with Azure Data Factory (ADF) and Azure Data Engineering ecosystem
Strong understanding of Data Modeling and Analytics data structures
Experience with Unity Catalog or Data Governance frameworks
Experience in performance tuning, query optimization, partitioning, and caching
Data Engineering experience
Azure Databricks experience
Valuable-to-Have Skills:
Git, Azure DevOps, GitHub Actions
CI/CD implementation and deployment automation
Terraform (Infrastructure as Code)
Docker and Kubernetes
Airflow
Kafka or Azure Event Hub
Power BI or Tableau
Databricks Workflows and Job Orchestration
Cluster Management and Optimization
Experience with Delta Table features such as MERGE, UPSERT, and Time Travel
BFSI domain experience
Key Responsibilities:
Design, develop, and optimize scalable batch and streaming data pipelines using Azure Databricks.
Develop data processing frameworks using PySpark and Spark SQL.
Implement Delta Lake-based storage solutions to ensure high reliability and data consistency.
Build and maintain Bronze, Silver, and Gold data layers within a Lakehouse architecture.
Ingest data from APIs, databases, Azure storage, and streaming platforms.
Develop scalable data models supporting analytics and re