Overview

We are looking for a

Senior Data Engineer (Databricks Lead)

with

12-16 years of experience to lead the design, development, and optimization of enterprise-scale data platforms. The ideal candidate will have deep expertise in

Databricks, Apache Spark, Snowflake, Kafka, Python, SQL, and AWS

, along with experience building scalable data architectures that support analytics, real-time processing, AI/ML, and Generative AI initiatives. .

Responsibilities

- Lead the architecture, design, and implementation of scalable, secure, and high-performance data platforms using Databricks, Snowflake, and AWS.

- Design and optimize large-scale batch and real-time data pipelines using Spark and Kafka.

- Develop and maintain robust ETL/ELT frameworks to ensure data reliability, scalability, and performance.

- Implement real-time data ingestion and processing solutions using Kafka and Spark Streaming.

- Build automated data integration frameworks leveraging REST APIs, GraphQL APIs, and Databricks APIs.

- Design cloud-native data solutions aligned with enterprise governance, security, and compliance standards.

- Develop data models using Star Schema, Snowflake Schema, Kimball, and Inmon methodologies.

- Provide technical leadership, architecture guidance, and mentorship to engineering teams.

- Collaborate with Analytics, Data Science, and AI teams to enable Machine Learning,



Generative AI, and LLM-based solutions.

- Troubleshoot and resolve platform performance bottlenecks and scalability challenges.

- Drive engineering best practices, code quality standards, and data governance initiatives.

Requirements

Technical Skills

- Strong expertise in Databricks and Apache Spark.

- Hands-on experience with Snowflake and modern data warehouse architectures.

- Proficiency in Kafka and real-time stream processing.

- Robust programming skills in Python and advanced SQL.

- Experience with AWS Cloud Services.

- Expertise in designing and implementing ETL/ELT architectures.

- Strong understanding of data modeling concepts and methodologies.

- Experience integrating systems using REST and GraphQL APIs.

Qualifications

- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or a related field.

- 12-16 years of experience in Data Engineering and Data Platform Architecture.

- Proven experience leading enterprise-scale data modernization and cloud migration initiatives.

Preferred Skills

- Experience with AI/ML data platforms, Generative AI, and LLM enablement.

- Exposure to Data Governance and Data Quality frameworks.

- Strong stakeholder management, communication, and technical leadership skills.

📌 Senior Data Engineer (Databricks Lead) (Bengaluru)
🏢 Prodapt
📍 Bengaluru