Job Summary We are seeking an experienced Data Engineering Lead / Architect with strong expertise in building enterprise-scale Lakehouse solutions using Azure, Databricks, PySpark, and modern cloud-native data engineering practices.
The ideal candidate should possess deep hands-on expertise in data engineering, semantic data modeling, metadata-driven architectures, and AI-assisted engineering approaches. This role requires a highly solution-oriented engineer who can go beyond pipeline development and contribute toward scalable architecture design, performance optimization, governance, automation, and business enablement.
The candidate should demonstrate strong ownership, problem-solving ability, adaptability to evolving priorities, and a continuous learning mindset toward emerging Azure and Databricks capabilities.
Key Responsibilities
Lakehouse Architecture & Engineering

Design and implement scalable end-to-end Lakehouse solutions using Azure, Databricks, Python, and PySpark.
Develop enterprise-grade data pipelines sourcing data from ERP, CRM, and field-device/on-field operational applications.
Implement Medallion Architecture (Bronze, Silver, Gold) with focus on scalability, governance, and maintainability.
Design robust Silver Layer semantic data models aligned with enterprise common data models.
Metadata & Semantic Layer Engineering

Develop and maintain semantic data layers including:
Data Catalogs
Data Dictionaries
Business Glossaries

Implement metadata-driven transformation and engineering frameworks.
Manage metadata lifecycle, lineage, and governance processes.




Handle schema drift and schema evolution through regenerative metadata and automated data dictionary approaches.
AI-Driven Automation & Productivity Engineering

Design AI-assisted solutions for extracting and generating metadata, data dictionaries, and semantic mappings using:
Source datasets
Application code
Defined business rules

Develop Databricks-based automation frameworks capable of dynamically generating transformation notebooks using metadata definitions.
Leverage AI capabilities to improve engineering productivity, automation, governance, and operational efficiency.
Continuously explore and adopt emerging Azure and Databricks AI capabilities.
Data Processing, Modeling & Optimization

Develop scalable data transformations, functions, procedures, and reusable business computation logic.
Build curated aggregates, summaries, snapshots, and analytics-ready datasets for downstream consumption.
Optimize distributed processing workloads for:
Performance
Scalability
Cost efficiency

~ Implement data quality frameworks including validation, reconciliation, observability, and governance controls.

Self-Service BI & Consumption Enablement

Enable governed Self-Service BI capabilities using Databricks Genie/API/embed tools.




Securely expose semantic/silver-layer datasets to business and functional stakeholders.
Capture and analyze usage telemetry/logs to continuously enhance data platform usability and adoption.
Required Skills & Expertise
Core Technical Skills

Strong expertise in:

SQL

Data Modeling
Python
PySpark
Azure
Databricks

Strong understanding of:
Medallion Architecture
Lakehouse Architecture
Metadata-driven engineering
Distributed data processing
Semantic data layers

Experience integrating data from:
ERP systems
CRM platforms
Field-device / operational applications

~ Experience with Databricks system tables and platform observability.

AI & Advanced Engineering Capabilities

Exposure to AI/ML-enabled data engineering solutions.
Experience leveraging AI capabilities for engineering productivity and automation.
Understanding of metadata automation and AI-assisted semantic enrichment approaches preferred.
Candidate Profile
We are looking for candidates who:

Possess strong analytical and problem-solving skills with ability to propose effective and scalable solutions.
Are flexible and adaptable to evolving business priorities and changing technical landscapes.
Demonstrate creativity and engineering innovation in solving enterprise data challenges.
Have solid communication and stakeholder management capabilities.
Are self-driven learners with continuous upskilling mindset on latest Azure and Databricks technologies.
Can operate beyond maintenance/support activities and contribute meaningfully toward solution architecture, optimization, and engineering transformation.

📌 Data Engineer SQL - Flexible Working (Bengaluru)
🏢 Tavant
📍 Bengaluru