Role Summary

We are hiring a Lead Data Engineer for a strongly Python-first profile, complemented by solid depth across Databricks, data modelling, T-SQL,

and data architecture. You will design, build, and own production-grade Python codebases and frameworks at the core of our Databricks

Lakehouse platform, while also setting technical direction on pipeline architecture, data modelling, and platform performance, and mentoring

a team of engineers.

Key Responsibilities

• Design, build, and own production-grade Python codebases for the data platform — ETL/ELT pipelines, reusable internal packages,

schema validation, and testing frameworks (pytest, CI/CD).

• Write clean, well-tested, performant Python: apply OOP, concurrency (multiprocessing/asyncio), decorators, and generators to solve real

pipeline-scale problems.

• Profile and optimize slow Python pipelines end to end — from algorithmic/code-level fixes to migrating heavy workloads onto

Databricks/Spark when Python alone won't scale.

• Set and enforce Python engineering standards across the team: code reviews,



packaging/versioning, structured logging, and exception

handling patterns.

• Design and deliver Bronze/Silver/Gold Medallion pipelines on Databricks — Delta Lake, Structured Streaming/Auto Loader, and Unity

Catalog governance.

• Own dimensional data modelling decisions — star/snowflake schema design and SCD strategy — for the platform's core data models.
• Write and review advanced T-SQL for platform workloads — window functions, MERGE/upsert logic, and execution-plan-based

performance tuning.

• Partner with Data Architects and stakeholders on ETL vs. ELT, orchestration, and cost/performance trade-offs across the platform; mentor

Data Engineers and Senior Data Engineers through design and code reviews.

Must-have Qualifications

8–12 years in data engineering, including at least 2 years in a technical leadership or mentoring capacity. This role carries a solid Python emphasis,

Complemented

📌 Lead Data Engineer/Associate Data Architect (Bengaluru)
🏢 Virtual Connect Solutions
📍 Bengaluru