Overview

We are looking for a Senior Lead - Databricks & Data Modernization to lead the design and implementation of a data modernization program involving both new data feature development in Databricks and the migration of existing capabilities from Palantir Foundry to Databricks.

The role requires a strong combination of Databricks engineering, data architecture, technical leadership and program execution skills. The candidate will be responsible for translating business and technical requirements into scalable Databricks solutions, guiding the engineering team, driving the Palantir-to-Databricks migration, and ensuring high-quality delivery.

The Senior Lead will work closely with customer stakeholders, enterprise/data architects, product owners, engineering teams and AI/ML teams and will be expected to take ownership of technical decisions and end-to-end delivery.

Responsibilities

- Technical Leadership & Solution Design
- Lead the design and development of scalable Databricks-based data solutions.
- Translate business requirements into technical architecture, data models and implementation plans.
- Define appropriate patterns for data ingestion, transformation, processing, storage and serving.
- Establish engineering standards, coding standards and reusable frameworks for the Databricks development team.
- Make technology and design decisions balancing performance, scalability, maintainability, security and cost.
- Review technical designs and code produced by the engineering team.
- Identify opportunities to modernize and simplify existing solutions.
- Palantir Foundry → Databricks Migration
- Lead the technical assessment and migration of existing Palantir Foundry capabilities to Databricks.
- Analyze existing Foundry pipelines, datasets, transformations, business rules and dependencies.
- Develop migration strategies and target-state architectures for different Foundry capabilities.
- Define appropriate Databricks-native replacements rather than simply performing a one-to-one migration.
- Identify migration complexity, dependencies, risks and technical gaps.
- Develop migration plans covering:
- Data ingestion
- Data transformation
- Data quality
- Data comparison/reconciliation
- Data models
- APIs/integrations
- Downstream analytics and applications

- Lead source-to-target mapping and functional/data validation.




- Establish approaches for achieving data and functional parity between Palantir and Databricks.
- Identify opportunities to improve performance, scalability and maintainability during migration.
- Databricks Engineering
- Provide hands-on technical leadership using:

- Databricks
- Apache Spark
- PySpark
- Spark SQL
- Python
- Delta Lake
- Unity Catalog
- Lakeflow / Spark Declarative Pipelines
- Databricks Workflows

- Design and implement batch and near-real-time data pipelines.
- Implement incremental processing, CDC and SCD patterns.
- Design and optimize Medallion Architecture implementations.
- Implement data quality, validation, reconciliation and error-handling frameworks.
- Optimize Spark workloads and Delta tables for performance and cost.
- Ensure production-grade engineering practices across the team.
- Data Architecture & Governance
- Work with enterprise architects to define the target Databricks data architecture.
- Define appropriate Bronze, Silver and Gold data layers.
- Establish data modelling and data integration standards.
- Ensure appropriate use of Unity Catalog for governance, access control and lineage.
- Define standards for data quality, metadata, lineage and observability.
- Ensure solutions comply with enterprise security and governance requirements.
- Team Leadership & Mentoring
- Lead and mentor a team of Databricks/data engineers.
- Break down requirements into technical tasks and assign work across the team.
- Provide technical guidance and unblock engineers on complex issues.
- Conduct design and code reviews.
- Establish engineering best practices and reusable components.
- Support capability development and knowledge sharing within the team.
- Act as the primary technical escalation point for the engineering team.
- Customer & Stakeholder Management
- Work directly with customer technical and business stakeholders.
- Lead technical discussions, requirement clarification and solution workshops.
- Present architecture, migration approaches and technical recommendations to senior stakeholders.
- Translate complex technical issues into clear business implications.




- Manage technical dependencies and communicate risks proactively.
- Work with customer teams to prioritize migration and recent feature development.
- Build strong working relationships with customer architecture, data and application teams.
- Delivery & Program Execution
- Own technical delivery for assigned workstreams.
- Develop technical delivery plans, estimates and milestones.
- Identify technical risks, dependencies and mitigation strategies.
- Track progress against technical deliverables.
- Ensure solutions meet agreed quality, performance and security standards.
- Coordinate with project/program managers to ensure successful delivery.
- Support release planning, production deployment and transition to operations.
- DevOps & Production Engineering
- Establish CI/CD practices for Databricks solutions.
- Implement Git-based development and deployment processes.
- Use Databricks Asset Bundles and appropriate deployment automation.
- Define automated testing strategies for data pipelines.
- Establish monitoring, logging and alerting standards.
- Lead root-cause analysis of production issues.
- Drive performance and cost optimization of Databricks workloads.

Requirements

Databricks - Mandatory

- Strong hands-on experience with Databricks
- Apache Spark / PySpark
- Spark SQL
- Delta Lake
- Unity Catalog
- Databricks Workflows / Jobs
- Lakeflow / Spark Declarative Pipelines
- Databricks Asset Bundles
- Databricks performance optimization
- Medallion Architecture

Data Engineering

- Large-scale data processing
- ETL/ELT architecture
- Data modelling
- Batch and incremental processing
- CDC
- SCD Type 1 / Type 2
- Data quality and reconciliation
- Data integration
- Structured and unstructured data processing

Programming

- Strong Python
- Strong SQL
- Good understanding of software engineering principles
- Experience developing reusable and production-grade frameworks

Cloud

Strong experience in Azure Cloud

Experience with cloud storage, networking, security and cloud data services is preferred.

DevOps

- Git
- CI/CD
- Automated testing
- Databricks deployment automation
- Infrastructure-as-code concepts
- Production monitoring and operational support

Hands-on experience with Palantir Foundry will be strongly preferred because the project involves migration of existing Foundry capabilities.

📌 AT&T Databricks - Lead (Gurugram)
🏢 Prodapt
📍 Gurugram