Job Responsibilities

- Lead the architecture, design, and implementation of enterprise-scale Databricks Data & AI platforms across AWS and Azure environments.

- Define and implement scalable Lakehouse architectures using Databricks, Delta Lake, and up-to-date data engineering patterns for batch and real-time workloads.

- Design high-performance data pipelines and transformation frameworks using Python, SQL, PySpark, Delta Live Tables, and Databricks Workflows/Jobs.

- Architect robust data ingestion, ETL/ELT, CDC, orchestration, and data processing solutions integrating enterprise, cloud, and streaming data sources.

- Establish enterprise-wide data governance, security, access control, and data discovery using Unity Catalog, including cataloging, lineage, permissions, and data-sharing strategies.

- Design and optimize Delta Lake architectures, including schema management, ACID transactions, partitioning, optimization, data lifecycle management, and performance tuning.

- Lead the implementation of ML and AI platforms using Databricks Mosaic AI and MLflow, supporting model development, experiment tracking, evaluation, deployment, and lifecycle management.

- Architect Generative AI solutions using LLMs, RAG architectures, Vector Search, embeddings, and enterprise data, ensuring scalable and production-ready AI applications.

- Design and implement Vector Search and Retrieval-Augmented Generation (RAG) solutions for enterprise knowledge discovery, intelligent search, and AI-powered applications.

- Establish responsible AI practices covering model governance, data security, evaluation, observability, hallucination mitigation, and access controls for enterprise GenAI workloads.





- Integrate Databricks with AWS/Azure cloud services, enterprise applications, databases, APIs, and external data platforms to build end-to-end data and AI solutions.

- Define standards for data modeling, pipeline development, CI/CD, DevOps, testing, monitoring, observability, and production operations across Databricks environments.

- Drive data platform modernization and migration initiatives, including transformation of legacy data warehouses and data lakes into scalable Databricks Lakehouse architectures.

- Optimize compute, storage, Spark workloads, SQL queries, Jobs, and Delta tables to improve performance, scalability, reliability, and cloud cost efficiency.

- Collaborate with data engineers, ML engineers, data scientists, cloud architects, security teams, and business stakeholders to translate business requirements into technical solutions.

- Lead architecture reviews, technical POCs, platform evaluations, and technology strategy initiatives related to Data Engineering, AI/ML, and GenAI.

- Troubleshoot complex data engineering, Spark, Delta Lake, ML, AI, security, and platform performance issues, providing architectural solutions and root-cause analysis.

- Provide technical leadership and mentorship to data engineers, ML engineers, and developers, establishing best practices and ensuring delivery against enterprise architecture standards.

Key Technical Skills

Databricks | Python | SQL | PySpark | AWS | Azure | Delta Lake | Unity Catalog | Databricks Workflows/Jobs | Delta Live Tables | Mosaic AI | MLflow | Vector Search | RAG | LLMs | GenAI | Lakehouse Architecture | Data Engineering | AI/ML | Data Governance | Cloud Data Platforms

📌 Databricks Data & AI Platform Architect (Hyderabad)
🏢 ZoomInData
📍 Hyderabad