17 Nov
|
Kadel Labs
|
India
Position: Databricks Architect
Number of Positions: 1
Experience: 10–12+ Years (with strong hands-on Databricks architecture experience)
Location: Udaipur, Jaipur, Bhopal, Bangalore
Employment Type: Full-Time
Role Overview
We are looking for an experienced Databricks Architect to lead the design, architecture, and implementation of enterprise-scale data platforms using the Databricks Lakehouse ecosystem. The ideal candidate should have strong expertise in Unity Catalog, Delta Lake, data governance, Lakehouse Monitoring, dimensional modeling, and platform cost optimization.
The candidate will be responsible for establishing scalable, secure, and cost-efficient data architectures while ensuring compliance with governance requirements, particularly for sensitive healthcare (PHI) and financial datasets. Strong stakeholder management and executive communication skills are essential.
Key Responsibilities
· Design and implement scalable enterprise data architectures using the Databricks Lakehouse platform.
· Define and implement data governance frameworks using Unity Catalog, including access controls, data lineage, metadata management, and auditing.
· Architect and optimize data storage and processing solutions using Delta Lake, including performance tuning and data reliability.
· Leverage Databricks System Tables for platform observability, usage analytics, workload monitoring, and operational insights.
· Implement and manage Lakehouse Monitoring to track data quality, data drift, anomalies, and pipeline health.
· Design enterprise data models, including dimensional modeling, Star Schema, Snowflake Schema, and fact/dimension structures.
· Establish Fin Ops practices for Databricks, including compute optimization, workload management, cluster utilization,
and cost monitoring.
· Define data security and compliance strategies for sensitive datasets, including PHI and financial data.
· Establish best practices for data access management, encryption, auditability, and regulatory compliance.
· Collaborate with engineering teams, business stakeholders, and senior leadership to translate business requirements into technical architecture.
· Provide architectural guidance, conduct design reviews, and mentor data engineering teams.
· Present architecture decisions, technical recommendations, platform performance, and cost optimization strategies to executive stakeholders.
Requirements Required Technical Skills
Skill Area
Required Expertise
Databricks
Strong enterprise-level Databricks architecture experience
Unity Catalog
Governance, access controls, lineage, metadata management
Delta Lake
Delta tables, optimization, performance tuning, reliability
System Tables
Platform monitoring, usage analytics, audit and cost insights
Lakehouse Monitoring
Data quality, anomaly detection, drift monitoring
Data Modeling
Dimensional modeling, Star Schema, Snowflake Schema
Fin Ops
Cost optimization, compute management, usage monitoring
Data Governance
Data security, privacy, compliance, access policies
Sensitive Data
PHI and financial data governance
Cloud Platforms
Azure / AWS / GCP with Databricks
Required Experience & Competencies
· Extensive experience in enterprise data architecture and data engineering.
· Proven experience designing and implementing Databricks-based Lakehouse solutions.
· Solid understanding of enterprise data governance, security, and compliance frameworks.
· Experience working with healthcare data, including PHI, and/or financial datasets.
· Ability to define architectural standards and guide large-scale data platform implementations.
· Strong communication skills with the ability to interact with C-level executives and business stakeholders.
· Ability to communicate complex technical concepts in business-oriented language.
Good to Have
· Databricks certifications such as Databricks Certified Data Engineer Professional or Databricks Certified Data Architect.
· Experience with Azure Data Factory, AWS Glue, or other data integration platforms.
· Knowledge of HIPAA and other data privacy and regulatory requirements.
· Experience with Power BI, Tableau, or other enterprise BI platforms.
· Experience leading enterprise-wide data modernization and cloud migration initiatives.
Key Deliverables
1. Enterprise Databricks Lakehouse architecture and implementation roadmap.
2. Unity Catalog-based enterprise data governance framework.
3. Optimized Delta Lake architecture and performance standards.
4. System Tables and Lakehouse Monitoring implementation.
5. Dimensional data modeling standards.
6. Databricks Fin Ops and cost optimization framework.
7. Security and compliance architecture for PHI and financial data.
8. Architecture documentation and executive-level technical presentations.
📌 Databricks Architect (India)
🏢 Kadel Labs
📍 India