Solution Architect – Databricks

Experience Required

- 10+ years of overall consulting / technology experience
- 7+ years of experience in Data Engineering, Data Platforms, Big Data, or Analytics
- Strong hands-on Databricks experience
- Minimum 6–8 end-to-end Databricks project implementations

Role Overview

We are looking for a highly experienced Senior FDE / Resident Solution Architect – Databricks who can work closely with enterprise customers to design, develop, optimize, and support scalable data engineering and analytics solutions on the Databricks platform.

The consultant should have strong hands-on expertise in Databricks, Apache Spark, PySpark, distributed computing, cloud platforms, performance optimization, and solution architecture.

This is a highly technical and client-facing role. The consultant should be able to independently drive architecture discussions, troubleshoot complex Databricks/Spark issues, provide implementation guidance, and support production deployments.

Key Responsibilities

- Design and implement scalable Databricks Lakehouse solutions.
- Work directly with customers to understand technical and business requirements.
- Define end-to-end data engineering and platform architecture.
- Build and optimize data pipelines using Databricks, Spark, PySpark, SQL, and Delta Lake.
- Design batch and streaming data-processing solutions.
- Provide technical guidance on Databricks architecture, development standards, and best practices.
- Troubleshoot complex Spark and Databricks performance issues.
- Optimize workloads for performance, scalability, reliability, and cost.
- Support enterprise Databricks platform implementation and modernization initiatives.
- Work with Databricks capabilities such as Delta Lake, Unity Catalog, Workflows, Auto Loader, Databricks SQL, Lakeflow/DLT, and Serverless.
- Design and support CI/CD processes for Databricks deployments.
- Work with DevOps and Infrastructure-as-Code tools such as Git, Terraform, Azure DevOps, GitHub, GitLab, or Jenkins.
- Provide guidance on Databricks security, governance, access control, and Unity Catalog.
- Support customer teams with architecture reviews, code reviews, troubleshooting, and technical mentoring.
- Work as a trusted technical advisor to customer architects, engineering teams,



and stakeholders.

Mandatory Skills

Databricks

- Strong hands-on experience in Databricks development and architecture
- Minimum 6–8 Databricks projects delivered
- Delta Lake
- Databricks Lakehouse Architecture
- Unity Catalog
- Databricks Workflows
- Auto Loader
- Databricks SQL
- Batch and streaming workloads
- Cluster / compute configuration
- Performance optimization

Apache Spark / PySpark

- Strong hands-on Spark and PySpark development
- Deep understanding of Spark internals including:
- Driver and Executors
- DAG
- Jobs, Stages, and Tasks
- Partitioning
- Shuffle
- Memory management
- Catalyst Optimizer
- Adaptive Query Execution
- Spark SQL execution plans
- Data skew
- Join optimization

Data Engineering

- Solid ETL / ELT experience
- Data ingestion and transformation
- Data pipelines
- Data modeling
- Batch processing
- Streaming processing
- SQL
- Python / PySpark
- Large-scale distributed data processing

Cloud

Candidate must have

- Deep expertise in at least one cloud platform: AWS / Azure / GCP
- Working knowledge of at least one additional cloud platform

Relevant cloud services may include: AWS: S3, IAM, Glue, Lambda, Kinesis, Redshift

Azure: ADLS, ADF, Key Vault, Entra ID, Synapse, Event Hubs

GCP: GCS, BigQuery, Pub/Sub, Dataflow, IAM

Performance & Scalability

Strong experience with

- Spark performance tuning
- Partitioning
- Shuffle optimization
- Data skew handling
- Join optimization
- Query optimization
- Delta table optimization
- Photon
- Cluster sizing
- Autoscaling
- Cost optimization

CI/CD & DevOps

Working knowledge of

- Git
- CI/CD pipelines
- Terraform
- Databricks Asset Bundles
- Azure DevOps / GitHub / GitLab / Jenkins
- Dev / Test / UAT / Production deployment processes

MLOps

Working knowledge of

- MLflow
- Experiment tracking
- Model Registry
- Model deployment / serving
- Model lifecycle management

Certification





Mandatory / Highly Preferred:

- Databricks Certified Data Engineer Professional
- Completion of relevant Databricks training/classes

Preferred Skills

- Enterprise Databricks architecture experience
- Databricks migrations / modernization
- Hadoop to Databricks migration
- Cloud data warehouse to Databricks migration
- Multi-cloud architecture exposure
- Unity Catalog implementation
- Terraform
- Databricks Asset Bundles
- MLflow / MLOps
- Data governance
- Streaming architecture
- Technical leadership / mentoring

Client-Facing Skills The consultant must have strong experience in:

- Customer-facing technical discussions
- Architecture workshops
- Requirement gathering
- Solution design
- Technical presentations
- Stakeholder management
- Architecture reviews
- Technical recommendations
- Troubleshooting and problem solving

Ideal Candidate Profile We are looking for candidates with approximately:

- 10–15+ years overall experience
- 7+ years Data Engineering / Big Data experience
- Strong Databricks and Spark experience
- 6–8+ Databricks project implementations
- Strong client-facing consulting background
- Databricks Data Engineer Professional certification
- Deep Spark / PySpark and Spark internals knowledge
- Strong performance tuning expertise
- Deep knowledge of one cloud platform and exposure to another
- Strong solution architecture and technical leadership capabilities

Vendor Submission Guidelines Please submit only candidates who meet the core requirements. Each submission should clearly mention:

- Total Experience
- Relevant Data Engineering Experience
- Databricks Experience
- Number of Databricks Projects Delivered
- Spark / PySpark Experience
- Primary Cloud Expertise
- Secondary Cloud Exposure
- Databricks Certifications
- Current Location
- Current CTC / Rate
- Expected CTC / Rate
- Notice Period / Availability
- Current Organization
- Client-facing / Consulting Experience
- Brief summary of the candidate's strongest Databricks project

Candidates with only basic Databricks notebook or ETL exposure should not be submitted.

- We are specifically looking for senior, hands-on Databricks professionals who can operate at a Solution Architect

📌 Solution Architect – Databricks (Bengaluru)
🏢 08World
📍 Bengaluru