Key Responsibilities
Data Pipeline Development
Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark.
Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
Develop and maintain data processing workflows using Delta Lake architecture.
Ensure productive data ingestion, transformation, and loading processes.
Data Management & Optimisation
Optimise performance of data pipelines and processing workloads.
Ensure data quality, reliability, and consistency across data platforms.
Write and optimise complex SQL queries for data processing and validation.
Implement data governance and access controls.
Cloud & Data Platform Integration
Work with Azure Data Lake for storing and managing large datasets.
Manage data cataloging and governance using Unity Catalog.
Integrate data workflows with CI/CD pipelines for automated deployments.
Support scalable and secure cloud-based data architectures.
Collaboration & Migration Projects
Work closely with cross-functional teams to deliver data-driven solutions.
Participate in data migration and modernisation initiatives.
Support troubleshooting and resolution of data pipeline issues.
Document data architecture, pipelines, and processes
Required Skills & Experience
Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks.
Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake.
Have a valuable understanding of RDBMS and proficiency in writing complex SQL (1st Preference Oracle / PLSQL).
Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
Optimise performance of data processing and ensure data quality standards.
Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines.
Solid knowledge of Databricks technologies, specifically Delta Lake and Apa
📌 Databricks Developer Hyderabad
🏢 Kumaran Systems
📍 Hyderabad
Reply to this offer
Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.