Key Responsibilities

- Design and develop scalable data pipelines using Azure and Databricks
- Build and maintain ETL/ELT workflows from diverse data sources
- Experience with Unity Catalog migration and data governance.
- Develop data transformations using Python, Spark, and Pandas
- Implement Delta Lake and Medallion architecture (Bronze/Silver/Gold layers)
- Manage data storage using Azure Data Lake Storage (ADLS)
- Design and optimize SQL queries and data models (star/dimensional schema)
- Implement CI/CD pipelines for data workflows using Azure DevOps or GitHub
- Ensure data quality, validation, monitoring, and observability
- Implement secure data access using Azure Entra ID (SSO/RBAC)
- Maintain transparent documentation for pipelines and data contracts
- Participate in Agile ceremonies, code reviews, and team collaboration
- Support and mentor junior team members

Required Skills & Experience

- 8+ years of experience in Data Engineering
- Strong experience with Azure Databricks (Python, Spark, Pandas)
- Experience with Azure Data Factory (ADF) for orchestration
- Hands-on experience with Delta Lake and Medallion architecture
- Strong knowledge of Azure Data Lake Storage (ADLS)




- Experience with Unity Catalog migration and data governance
- Solid proficiency in SQL and performance optimization
- Experience in ETL/ELT and data modeling (star/dimensional schema)
- Experience with CI/CD tools (Azure DevOps or GitHub Enterprise)
- Understanding of Azure Entra ID for security and access control
- Experience in data validation, testing, and monitoring pipelines
- Familiarity with Agile/Scrum methodologies (Jira/Confluence)
- Strong communication, documentation, and collaboration skills

Preferred Experience :

- Experience with Databricks DevOps (cluster configuration, secrets management)
- Experience with Azure Functions (Python or C#)
- Exposure to Synapse SQL pools, dbt, or Delta Live Tables
- Experience in Financial Services domain
- Ability to work independently and contribute to team outcomes
- Strong focus on secure, production-grade data solutions
- Commitment to data reliability, performance, and governance
- Continuous learning mindset and adaptability
- Responsible use of Generative AI tools with validation and guardrails

Qualification

Any degree relevant to IT or related field

📌 Lead Data Engineer (Hyderabad)
🏢 Kumaran Systems
📍 Hyderabad