Data Engineer/Developer

Role Summary

Design, build, and optimize Azure data pipelines and lakehouse solutions. Deliver secure, reliable datasets with solid governance, automation, and documentation. Collaborate across teams and contribute to standards in an Agile setting.

Must-Have (Day 1)
Experience: 4–6 years in data engineering
Core Platform: Databricks with Python, Spark, Pandas (notebooks and modular code)
Orchestration: Azure Data Factory (pipelines, integration runtimes); ingest from diverse sources
Lakehouse: Delta Lake fundamentals; Medallion architecture (bronze/silver/gold) in production
Storage/SQL/Performance: Azure Data Lake Storage (ADLS); solid SQL; performance-aware design
Data Patterns: ETL/ELT; data modeling (e.g., dimensional/star schema)
DevOps & Security: CI/CD for data projects (Azure DevOps or GitHub Enterprise); familiarity with Azure Entra ID for SSO/RBAC; secure workspace/data access
Quality & Observability: Data validation/testing, code reviews, and basic monitoring/alerting for jobs/pipelines
Ways of Working: Agile/Scrum (Jira/Confluence); clear pipeline and data contract documentation
Collaboration:



Effective stakeholder engagement; support/mentor junior team members; clear communication
Generative AI (Day 1)
Prompt design for data tasks (ingestion, transformations, documentation) with clear objectives and constraints
Use of Copilot/ChatGPT to scaffold notebooks/jobs, generate tests, and optimize SQL/Spark—validates outputs before merging

Nice-to-Have (Train within 60–90 days)
Unity Catalog migration (Hive to Unity) and permissions/governance
Databricks DevOps (cluster configuration, secret management, workspace automation)
Azure Functions (C# or Python) for orchestration/integration
Synapse dedicated SQL pools or dbt; Delta Live Tables
Financial services domain exposure

Shared Expectations
Work independently with minimal supervision while contributing to team outcomes
Commitment to secure practices and production-grade reliability
Continuous improvement mindset and willingness to learn new tools/technologies
Willingness to work within regulated environment controls and policies
Use Generative AI responsibly to improve velocity and quality (straightforward, structured prompts; guardrails; validate AI-assisted outputs before adoption)

📌 Data Engineer Hyderabad
🏢 Kumaran Systems
📍 Hyderabad