Bengaluru, Karnataka
Job Summary

Job Summary

We are seeking a highly experienced, strategic, and hands-on Lead Data Engineer to join our enterprise data engineering team. With over 12 years of professional expertise, you will act as a bridge between complex business requirements and high-performance technical solutions.

In this role, you will architect resilient, cloud-agnostic Databricks and Lakehouse environments, design enterprise-grade streaming and batch ingestion pipelines, and implement modern architectures such as Data Mesh. You will also drive innovation by integrating Generative AI capabilities (including RAG frameworks and LLM fine-tuning) into up-to-date data platforms, while establishing robust, enterprise-wide data governance and compliance.

Key Responsibilities

1. Architecture & Lakehouse Design

Architect and optimize highly scalable, resilient cloud-agnostic Databricks Lakehouse environments to convert multi-terabyte batch and streaming data into actionable intelligence.
Lead complex, large-scale database and process migrations from legacy estates (e.g., SQL Server, on-premise systems) to modern cloud environments (Azure, AWS, Snowflake, MS Fabric), improving query performance and reducing infrastructure overhead.
Design and deploy modern semantic data models and high-performance BI reporting solutions (such as Power BI or Tableau) to enable executives with real-time KPI insights.

2. Data Pipeline & API Engineering (ETL/ELT)

Design, build, and dynamically parameterize reusable ETL/ELT pipelines using platforms like Azure Data Factory (ADF) , AWS Glue/Lambda/Step Functions , Snaplogic , Ab Initio , or Microsoft Fabric .
Develop robust stream-processing pipelines utilizing Apache Kafka ,



Azure Event Hubs , or Amazon Event Bridge alongside REST APIs to build synchronous and event-driven data workflows.
Implement automated data-loading accelerators (such as Databricks Auto Loader) to ingest wide-ranging, heterogeneous data structures (structured, semi-structured, APIs, and flat files) with minimal manual overhead.

3. Data Governance, Compliance & Security

Devise and enforce unified data governance frameworks incorporating data quality (DQ) metrics, Attribute-Based Access Control (ABAC), data lineage, and compliance tracking.
Ensure rigorous protection of PII and adherence to international regulatory standards (e.g., ISO, GDPR, HIPAA) using tools like Microsoft Purview or Informatica MDM/IDD .

4. Generative AI & Innovation

Lead the adoption of emerging AI technologies by designing RAG (Retrieval-Augmented Generation) virtual assistants and integrating them natively into Databricks platforms.
Collaborate on fine-tuning Large Language Models (LLMs) on domain-specific datasets to deliver high-quality, context-aware responses in real time.

5. Leadership & Project Delivery

Provide technical leadership and mentorship to cross-functional, onshore-offshore engineering teams.
Manage stakeholder relationships, collaborating with business users, functional analysts,



and executive teams to scope and prioritize key deliverables.

Enforce agile practices (Scrum/Kanban) and DevOps standards (CI/CD pipelines via Azure DevOps/GitHub) to maintain delivery excellence

Skill Requirements

Required Technical Skills:
Cloud & ETL/ELT: Deep expertise in Azure (ADF, Azure Synapse, ADLS), Azure Databricks, MS Fabric, AWS (S3, EMR, DMS, Glue, Lambda), Snowflake, Snaplogic, or SSIS.
Big Data & Databases: Strong experience in Postgres, Oracle, SQL Server, Snowflake, Teradata, MongoDB, and Hadoop/Hive ecosystems.
Programming & Scripting: Mastery in Python , PySpark , T-SQL , and Unix Shell Scripting. Experience in C# or Java is highly desirable.
Streaming & Integration: Apache Kafka, Azure Event Hubs, REST API integration, and Change Data Capture (CDC) technologies (e.g., Qlik Replicate).
AI/BI Tools: Power BI, Tableau, RAG implementations, and LLM orchestration.
DevOps & Tools: Azure DevOps, GitHub, Jira, and SFTP automation.

Qualifications & Education:
Education: Bachelor’s Degree in Engineering (B.E./B.Tech) in Computer Science, Information Technology, or a closely related field.

Certifications (Highly Valued):
Databricks Certified Data Engineer Associate / Professional
Databricks Generative AI Fundamentals or Lakehouse Fundamentals
Scrum Fundamentals Certified (SFC)
Snaplogic or cloud provider certifications (Azure/AWS)

Other Requirements
#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-#body.unify div.unify-button-container .unify-apply-now: focus, #body.unify div.unify-button-container .unify-apply-

📌 Senior Technical Lead (India)
🏢 HCLTech
📍 India