14 Nov
|
Talentmatics
|
India
14 Nov
Talentmatics
India
Our client is looking for a skilled Data Engineer – Databricks with experience in data engineering and solid hands-on expertise in Azure Databricks, PySpark, SQL, and Delta Lake. The candidate will be responsible for developing, optimizing, and maintaining scalable data pipelines and implementing reliable data processing solutions.
Key Responsibilities
Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and SQL.
Build and manage data ingestion and transformation processes using Delta Lake and Delta Tables.
Implement Auto Loader (cloudFiles), schema inference, and schema evolution.
Develop and manage Spark Declarative Pipelines (SDP/DLT).
Configure and manage Databricks Workflows, Jobs, Tasks, and Triggers.
Implement data governance, security, and access management using Unity Catalog.
Design dimensional data models, data marts, and Slowly Changing Dimensions (SCD).
Develop and optimize Databricks SQL Dashboards and work with AI/BI, Genie, or similar BI tools.
Monitor and troubleshoot data pipelines to ensure performance, reliability, and scalability.
Requirements
6–8 years of overall experience in Data Engineering.
Solid hands-on experience with Azure Databricks, PySpark, and SQL.
Expertise in Delta Lake and Delta Tables.
Practical experience with Auto Loader (cloudFiles), schema inference, and schema evolution.
Hands-on experience with Spark Declarative Pipelines (SDP/DLT).
Good understanding of Databricks Workflows, Jobs, Tasks, and Triggers.
Solid knowledge of Unity Catalog, data governance, and access management.
Experience with dimensional modelling, data marts, and SCD.
Familiarity with Databricks SQL Dashboards, AI/BI, and Genie.
Working knowledge of cloud platforms such as Azure, AWS, or GCP.
Qualifications
Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline.
Strong understanding of data engineering concepts, ETL/ELT processes, and data pipeline architecture.
Excellent analytical, problem-solving, and troubleshooting skills.
Ability to work independently and collaborate effectively with cross-functional teams.
📌 Data Engineer – Databricks Pune (India)
🏢 Talentmatics
📍 India