22 Nov
|
Hunterz Research
|
Bengaluru
22 Nov
Hunterz Research
Bengaluru
Senior Databricks ArchitectWork Location: Bangalore - Pune - Noida - Hyderabad - ChennaiNotice period: Immediate and Max 15 to 20 DaysExperience: 10 - 15 yearsRole Overview:Senior platform and solution architect responsible for enterprise-scale Databricks lakehouse architecture, modernization strategies, governance, migration roadmaps, engineering standards, performance and cost guardrails, and AI-ready data foundations. The role converts business and non-functional requirements into implementable platform patterns and guides engineering teams through delivery.Core Responsibilities:Lakehouse & Platform Architecture:- Define target-state lakehouse architecture, workspace and setting strategy, Unity Catalog topology, storage and compute patterns, governed sharing, lineage and domain boundaries.- Establish implementation standards for Medallion architecture, Lakeflow Declarative Pipelines / DLT, Workflows, Auto Loader, Delta Lake, system tables and Databricks Asset Bundles.- Create reference architectures, architecture decision records, reusable blueprints and non-functional requirement guardrails.Spark, Data Engineering & Streaming:- Set engineering standards for PySpark, Spark SQL, Delta Lake, Structured Streaming, CDC, SCD, incremental processing, data contracts and schema evolution.- Guide execution-plan analysis, AQE, join strategy, shuffle and skew management, partitioning, file sizing, caching and Photon adoption.- Drive idempotent, auditable and recoverable pipelines with clear batch-versus-streaming decision rationale.Migration & Modernization:- Lead legacy warehouse, Hadoop and ETL modernization assessments and define migration waves, conversion patterns and dependency plans.- Own validation, reconciliation, parallel-run, cutover, rollback and decommissioning approaches.- Create reusable migration playbooks and accelerators while controlling delivery risk and business continuity.Governance,
Security & Data Sharing:- Design Unity Catalog operating patterns covering catalogs, schemas, external locations, service principals, RBAC/ABAC, masking, row/column controls and auditability.- Define lineage, data quality, privacy, retention and compliance-ready implementation standards.- Guide Delta Sharing, Lakehouse Federation and governed producer-consumer patterns where appropriate.Platform Engineering, DataOps & Reliability:- Define Git-based CI/CD, testing, environment promotion, infrastructure-as-code and release controls for Databricks assets.- Establish SLIs/SLOs, observability, alerting, runbooks, reliability scorecards and production-readiness reviews using platform telemetry and system tables.- Improve developer experience through templates, reusable frameworks, guardrails and self-service patterns.Cost Optimization & FinOps:- Establish cluster and compute policies, autoscaling, workload sizing, serverless decision guidelines, tagging and chargeback/showback controls.- Create cost and performance baselines and drive workload-level optimization with measurable outcomes.- Balance performance, reliability and cost rather than optimizing one dimension in isolation.AI-ready Data Foundations:- Guide governed data foundations for RAG and AI use cases, including data quality, lineage, access controls and retrieval readiness.- Demonstrate working awareness of Vector Search, model serving/lifecycle, MLflow, context engineering and AI governance.- Assess emerging capabilities without weakening core data engineering and platform-accountability expectations.Consulting & Stakeholder Leadership:- Lead discovery and architecture workshops; articulate options, trade-offs, risks, roadmap and estimates to technical and business stakeholders.- Explain the latest project with clear business problem, architecture choice, Databricks fit, alternatives considered, challenges, outcomes and lessons learned.- Mentor engineers and leads, review designs, and support proposals, solution narratives and customer-facing architecture discussions. (ref:hirist.tech)
📌 Senior Databricks Architect (Bengaluru)
🏢 Hunterz Research
📍 Bengaluru