19 Nov
|
Prodapt
|
Bengaluru
Overview
Prodapt is seeking a highly experienced Senior Databricks Engineer to lead the design and implementation of modern cloud-based data platforms. The ideal candidate will have deep expertise in Databricks, Snowflake, AWS, Data Engineering, and Enterprise Data Architecture, with proven experience building scalable, secure, and high-performance data solutions. This role requires strong technical leadership, hands-on architecture experience, and the ability to drive enterprise-wide data modernization initiatives while supporting advanced analytics, AI, and machine learning workloads.
Responsibilities
- Design and architect scalable, secure, and highly available enterprise data platforms using Databricks, Snowflake, and AWS Cloud.
- Lead the development and optimization of large-scale ETL/ELT pipelines, supporting both batch and real-time data processing workloads.Design and implement modern data lakehouse architectures and cloud-native data ecosystems.
- Architect and deploy AWS-based data platform components while ensuring compliance with enterprise security, IAM, governance, and networking standards.Build and optimize data ingestion frameworks integrating Databricks APIs, REST APIs, GraphQL APIs, and third-party data sources.
- Develop real-time data processing solutions for business-critical, low-latency use cases using modern streaming technologies.
- Design and implement scalable data models, incorporating both traditional data warehousing methodologies (Kimball, Inmon, Star Schema, Snowflake Schema) and modern semi-structured/unstructured data patterns.
- Collaborate with business stakeholders, architects,
data scientists, and engineering teams to define and execute data strategies.Drive performance optimization, cost management, scalability, and reliability initiatives across the data platform landscape.
- Provide technical leadership, architectural governance, code reviews, and mentorship to engineering teams.
- Support AI and Machine Learning initiatives by designing data foundations that enable LLMs, Generative AI, Machine Learning pipelines, and advanced analytics solutions.
- Establish best practices for data engineering, architecture, governance, observability, and platform operations.Prepare and maintain architecture documentation, design standards, and technical roadmaps.
.
Requirements
- 12 to 16 years of experience in Data Engineering, Data Architecture, and Cloud Data Platforms.
- Strong expertise in Databricks, Snowflake, and AWS Cloud Services.
- Proven experience designing and delivering large-scale enterprise data architectures and contemporary data lakehouse solutions.
- Hands-on experience building and optimizing ETL/ELT pipelines, data ingestion frameworks, and distributed data processing systems.
- Strong knowledge of real-time and batch data processing architectures.
- Expertise in Databricks Workflows, Delta Lake, Spark, Structured Streaming, and Databricks APIs.
- Experience integrating REST and GraphQL APIs for enterprise-scale data ingestion and orchestration.
- Strong understanding of Data Warehousing concepts including Kimball, Inmon, Star Schema, and Snowflake Schema methodologies.
- Experience designing scalable and secure solutions within the AWS ecosystem, including IAM, networking, and governance controls.
- Strong proficiency in SQL, Python, and Spark-based development.
- Experience with performance tuning, optimization, data quality, metadata management, and platform observability.
- Ability to provide technical leadership, solution architecture guidance, and mentorship to engineering teams.
- Excellent stakeholder management, communication, problem-solving, and decision-making skills.
Preferred Qualifications
Experience with modern data ecosystem tools such as Apache Kafka, Airflow, dbt, Iceberg, and Kubernetes. Exposure to Generative AI, LLMs, Machine Learning platforms, MLOps, and advanced analytics architectures will be highly preferred.
Experience with data governance, security, compliance frameworks, and enterprise-scale cloud modernization programs is advantageous. Relevant certifications in Databricks, Snowflake, AWS Solutions Architecture, or Data Engineering will be considered a strong advantage. Candidates with experience supporting AI-ready data platforms and leading enterprise data transformation initiatives will be highly valued.
📌 Senior Databricks Engineer (Bengaluru)
🏢 Prodapt
📍 Bengaluru