We are seeking a skilled Data Engineer to design, build, and optimize Azure-based data pipelines and lakehouse solutions. The ideal candidate will deliver secure, reliable datasets with strong governance, automation, and documentation while collaborating across teams in an Agile environment. This role requires solid expertise in Databricks, Python, Spark, data pipelines, and Azure ecosystem, along with a focus on performance, scalability, and data quality.Key ResponsibilitiesDesign and develop scalable data pipelines using Azure and DatabricksBuild and maintain ETL/ELT workflows from diverse data sourcesExperience with Unity Catalog migration and data governance.Develop data transformations using Python, Spark, and PandasImplement Delta Lake and Medallion architecture (Bronze/Silver/Gold layers)Manage data storage using Azure Data Lake Storage (ADLS)Design and optimize SQL queries and data models (star/dimensional schema)Implement CI/CD pipelines for data workflows using Azure DevOps or GitHubEnsure data quality, validation, monitoring, and observabilityImplement secure data access using Azure Entra ID (SSO/RBAC)Maintain clear documentation for pipelines and data contractsParticipate in Agile ceremonies, code reviews,



and team collaborationSupport and mentor junior team membersRequired Skills & Experience 8+ years of experience in Data EngineeringStrong experience with Azure Databricks (Python, Spark, Pandas)Experience with Azure Data Factory (ADF) for orchestrationHands-on experience with Delta Lake and Medallion architectureStrong knowledge of Azure Data Lake Storage (ADLS)Experience with Unity Catalog migration and data governanceStrong proficiency in SQL and performance optimizationExperience in ETL/ELT and data modeling (star/dimensional schema)Experience with CI/CD tools (Azure DevOps or GitHub Enterprise)Understanding of Azure Entra ID for security and access controlExperience in data validation, testing, and monitoring pipelinesFamiliarity with Agile/Scrum methodologies (Jira/Confluence)Strong communication, documentation, and collaboration skillsPreferred Experience :Experience with Databricks DevOps (cluster configuration, secrets management)Experience with Azure Functions (Python or C#)Exposure to Synapse SQL pools, dbt, or Delta Live TablesExperience in Financial Services domainAbility to work independently and contribute to team outcomesStrong focus on secure, production-grade data solutionsCommitment to data reliability, performance, and governanceContinuous learning mindset and adaptability

📌 Data Engineer (Chennai)
🏢 Kumaran Systems
📍 Chennai