We are seeking a highly experienced Lead Data Engineer to lead the design, development, and delivery of scalable, reliable, and cost-effective data platforms. The ideal candidate will possess deep expertise in modern data engineering technologies, including distributed data processing, ETL/ELT pipeline development, data modeling, and workflow orchestration, along with hands-on experience with the Databricks Lakehouse Platform and its medallion (Bronze/Silver/Gold) architecture.
This role also requires a solid understanding of Large Language Models (LLMs) and GenAI data ecosystems, enabling the development of high-quality datasets and retrieval-ready pipelines for AI-powered applications. As a technical leader, you will mentor engineering teams, establish best practices, and collaborate closely with architects, data scientists, and business stakeholders to deliver robust data solutions.
Key Responsibilities
Lead the architecture, development, and optimization of scalable data platforms supporting both batch and streaming workloads.
Design, implement,



and maintain ETL/ELT pipelines for data ingestion, transformation, and curation using the Databricks Lakehouse Platform and medallion architecture.
Define data models, schemas, and storage strategies for data lakes and data warehouses to support analytics, reporting, machine learning, and GenAI initiatives.
Develop and curate high-quality datasets, feature engineering pipelines, and retrieval-ready data stores, including embeddings and vector-based data structures, for LLM-powered applications.
Establish engineering standards, coding best practices, code review processes, and CI/CD pipelines to ensure maintainable and reliable solutions.
Build and automate end-to-end data workflows using orchestration tools such as Apache Airflow or equivalent platforms.
Lead migrations from legacy on-premises or cloud-based data warehouses to modern cloud-native and lakehouse architectures.
Optimize performance and cost by implementing

📌 Lead Data Engineer (Bengaluru)
🏢 Trantor
📍 Bengaluru