Role Summary

Lead a Python-first data engineering function, owning production-grade codebases and frameworks for a Databricks Lakehouse platform, while setting direction for pipeline architecture, data modelling and performance.

Key Responsibilities

- Build production Python ETL/ELT pipelines, reusable packages, schema validation and testing frameworks (pytest, CI/CD).

- Write performant Python using OOP, generators, decorators and concurrency (multiprocessing/asyncio); profile, debug and optimize pipeline code.

- Set Python engineering standards for reviews, packaging/versioning, logging and exception handling.

- Design Bronze/Silver/Gold Medallion pipelines on Databricks using Delta Lake, Structured Streaming, Auto Loader and Unity Catalog.

- Own dimensional modelling (star/snowflake, SCD); write/review advanced T-SQL, MERGE/upsert logic and execution-plan-based tuning.

- Partner on ETL/ELT, orchestration and cost/performance decisions; mentor Data Engineers and Senior Data Engineers.

Must-Have Qualifications

- 8–12 years in data engineering, including 2+ years in technical leadership or mentoring.

- Strong Python: OOP, generators, decorators, concurrency, exception handling, pytest and production pipeline frameworks.





- Hands-on Databricks: Delta Lake (ACID, time travel, OPTIMIZE/ZORDER/VACUUM), Structured Streaming, Auto Loader and Unity Catalog.

- Spark performance tuning, data skew, cluster sizing/autoscaling and cost optimization.

- Dimensional modelling: star/snowflake, SCD Types 1/2/3, fact/dimension grain and conformed dimensions; working knowledge of Data Vault and

Kimball/Inmon.

- Advanced T-SQL, indexing and query tuning; Medallion architecture and ETL/ELT and batch/streaming trade-offs.

- Orchestration using Databricks Workflows, Airflow or ADF on Azure or AWS.

Good to Have & Education

- PCEP/PCAP or equivalent Python certification; Databricks Certified Data Engineer Qualified.

- Data governance/compliance (GDPR/HIPAA), FinOps and experience leading/mentoring 3+ engineers.

- Bachelor’s/Master’s in Computer Science, Engineering or related field, or equivalent practical experience.

- Strong stakeholder communication and ability to explain technical trade-offs and own enterprise-scale design decisions

Pay: ₹1,500,000.00 - ₹2,100,000.00 per year

Work Location: In person

📌 Lead Data Engineer (Bengaluru)
🏢 Kalpita Technologies
📍 Bengaluru