Job Description
We are seeking a skilled Data Engineer to design, build, and optimize Azure-based data pipelines and lakehouse solutions. The ideal candidate will deliver secure, reliable datasets with strong governance, automation, and documentation while collaborating across teams in an Agile environment. This role requires robust expertise in Databricks, Python, Spark, data pipelines, and Azure ecosystem, along with a focus on performance, scalability, and data quality.
n

nKey Responsibilities
n
n
Design and develop scalable data pipelines using Azure and Databricksn
Build and maintain ETL/ELT workflows from diverse data sourcesn
Experience with Unity Catalog migration and data governance.n
Develop data transformations using Python, Spark, and Pandasn
Implement Delta Lake and Medallion architecture (Bronze/Silver/Gold layers)n
Manage data storage using Azure Data Lake Storage (ADLS)n
Design and optimize SQL queries and data models (star/dimensional schema)n
Implement CI/CD pipelines for data workflows using Azure DevOps or GitHubn




Ensure data quality, validation, monitoring, and observabilityn
Implement secure data access using Azure Entra ID (SSO/RBAC)n
Maintain transparent documentation for pipelines and data contractsn
Participate in Agile ceremonies, code reviews, and team collaborationn
Support and mentor junior team membersn

n

nRequired Skills & Experience
n
n
8+ years of experience in Data Engineeringn
Strong experience with Azure Databricks (Python, Spark, Pandas)n
Experience with Azure Data Factory (ADF) for orchestrationn
Hands-on experience with Delta Lake and Medallion architecturen
Robust knowledge of Azure Data Lake Storage (ADLS)n
Experience with Unity Catalog migration and data governancen
Strong proficiency in SQL and performance optimizationn
Experience in ETL/ELT and data modeling (star/dimensional schema)n
Experience with CI/CD tools (Azure DevOps or GitHub Enterprise)n

📌 Data Engineer Tamil Nadu
🏢 Kumaran Systems
📍 Tamil Nadu