17 Nov
|
Kumaran Systems
|
Chennai
17 Nov
Kumaran Systems
Chennai
Key Responsibilities
Data Pipeline Development
• Design and develop scalable data pipelines using Azure Databricks, Python, and PySpark.
• Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
• Develop and maintain data processing workflows using Delta Lake architecture.
• Ensure effective data ingestion, transformation, and loading processes. Data Management & Optimisation
• Optimise performance of data pipelines and processing workloads.
• Ensure data quality, reliability, and consistency across data platforms.
• Write and optimise complex SQL queries for data processing and validation.
• Implement data governance and access controls. Cloud & Data Platform Integration
• Work with Azure Data Lake for storing and managing large datasets.
• Manage data cataloging and governance using Unity Catalog.
• Support scalable and secure cloud-based data architectures. Collaboration & Migration Projects
• Work closely with cross-functional teams to deliver data-driven solutions.
• Participate in data migration and modernisation initiatives.
• Support troubleshooting and resolution of data pipeline issues.
• Document data architecture, pipelines, and processes. Required Skills & Experience
· Must have 7 years of IT experience and at least 5+ years of hands-on experience in Databricks with good communication skills.
· Design and develop scalable data pipelines using Azure Databricks, SCALA, Python, PySpark, and Delta Lake.
· Have a good understanding of RDBMS and proficiency in writing complex SQL (1st Preference – Oracle / PLSQL).
· Implement ETL/ELT workflows for structured, semi-structured, and unstructured data.
· Optimise performance of data processing and ensure data quality standards.
· Work with Azure Data Lake, Unity Catalogue, and integrate with CI/CD pipelines.
· Experience with orchestration tools like Airflow or Azure Data Factory is a plus.
· Have working experience on a migration project.
· Preferable to have experience in Informatica (although not mandatory) Relevant Experience Preferred
• Hands-on experience developing data pipelines using Azure Databricks and PySpark.
• Experience implementing ETL/ELT solutions for enterprise data platforms.
• Experience optimizing data processing performance and data workflows.
• Experience working with Azure Data Lake and cloud-based data architectures.
• Experience working on data migration or modernization projects.
📌 Senior Data engineer (Chennai)
🏢 Kumaran Systems
📍 Chennai