Overview We are looking for a Senior Lead - Databricks & Data Modernization to lead the design and implementation of a data modernization program involving both new data feature development in Databricks and the migration of existing capabilities from Palantir Foundry to Databricks .

The role requires a solid combination of Databricks engineering, data architecture, technical leadership and program execution skills . The candidate will be responsible for translating business and technical requirements into scalable Databricks solutions, guiding the engineering team, driving the Palantir-to-Databricks migration, and ensuring high-quality delivery.

The Senior Lead will work closely with customer stakeholders, enterprise/data architects, product owners, engineering teams and AI/ML teams and will be expected to take ownership of technical decisions and end-to-end delivery.

Responsibilities

- Technical Leadership & Solution Design
- Lead the design and development of scalable Databricks-based data solutions.
- Translate business requirements into technical architecture, data models and implementation plans.
- Define appropriate patterns for data ingestion, transformation, processing, storage and serving.
- Establish engineering standards, coding standards and reusable frameworks for the Databricks development team.
- Make technology and design decisions balancing performance, scalability, maintainability, security and cost.
- Review technical designs and code produced by the engineering team.
- Identify opportunities to modernize and simplify existing solutions.
- Palantir Foundry → Databricks Migration
- Lead the technical assessment and migration of existing Palantir Foundry capabilities to Databricks.
- Analyze existing Foundry pipelines, datasets, transformations, business rules and dependencies.
- Develop migration strategies and target-state architectures for different Foundry capabilities.
- Define appropriate Databricks-native replacements rather than simply performing a one-to-one migration.
- Identify migration complexity, dependencies, risks and technical gaps.

- Develop migration plans covering

- Data ingestion

- Data transformation

- Data quality

- Data comparison/reconciliation

- Data models

- APIs/integrations

- Downstream analytics and applications
- Lead source-to-target mapping and functional/data validation.




- Establish approaches for achieving data and functional parity between Palantir and Databricks.
- Identify opportunities to improve performance, scalability and maintainability during migration.

- Databricks Engineering
- Provide hands-on technical leadership using:

- Databricks

- Apache Spark

- PySpark

- Spark SQL

- Python

- Delta Lake

- Unity Catalog
- Lakeflow / Spark Declarative Pipelines

- Databricks Workflows
- Design and implement batch and near-real-time data pipelines.
- Implement incremental processing, CDC and SCD patterns.
- Design and optimize Medallion Architecture implementations.
- Implement data quality, validation, reconciliation and error-handling frameworks.
- Optimize Spark workloads and Delta tables for performance and cost.
- Ensure production-grade engineering practices across the team.
- Data Architecture & Governance
- Work with enterprise architects to define the target Databricks data architecture.
- Define appropriate Bronze, Silver and Gold data layers.
- Establish data modelling and data integration standards.
- Ensure appropriate use of Unity Catalog for governance, access control and lineage.
- Define standards for data quality, metadata, lineage and observability.
- Ensure solutions comply with enterprise security and governance requirements.
- Team Leadership & Mentoring
- Lead and mentor a team of Databricks/data engineers.
- Break down requirements into technical tasks and assign work across the team.
- Provide technical guidance and unblock engineers on complex issues.
- Conduct design and code reviews.
- Establish engineering best practices and reusable components.
- Support capability development and knowledge sharing within the team.
- Act as the primary technical escalation point for the engineering team.
- Customer & Stakeholder Management
- Work directly with customer technical and business stakeholders.
- Lead technical discussions, requirement clarification and solution workshops.
- Present architecture, migration approaches and technical recommendations to senior stakeholders.




- Translate complex technical issues into clear business implications.
- Manage technical dependencies and communicate risks proactively.
- Work with customer teams to prioritize migration and new feature development.
- Build strong working relationships with customer architecture, data and application teams.
- Delivery & Program Execution
- Own technical delivery for assigned workstreams.
- Develop technical delivery plans, estimates and milestones.
- Identify technical risks, dependencies and mitigation strategies.
- Track progress against technical deliverables.
- Ensure solutions meet agreed quality, performance and security standards.
- Coordinate with project/program managers to ensure successful delivery.
- Support release planning, production deployment and transition to operations.
- DevOps & Production Engineering
- Establish CI/CD practices for Databricks solutions.
- Implement Git-based development and deployment processes.
- Use Databricks Asset Bundles and appropriate deployment automation.
- Define automated testing strategies for data pipelines.
- Establish monitoring, logging and alerting standards.
- Lead root-cause analysis of production issues.
- Drive performance and cost optimization of Databricks workloads.

Requirements Databricks - Mandatory

- Strong hands-on experience with Databricks
- Apache Spark / PySpark
- Spark SQL
- Delta Lake
- Unity Catalog
- Databricks Workflows / Jobs
- Lakeflow / Spark Declarative Pipelines
- Databricks Asset Bundles
- Databricks performance optimization
- Medallion Architecture

Data Engineering
- Large-scale data processing
- ETL/ELT architecture
- Data modelling
- Batch and incremental processing
- CDC
- SCD Type 1 / Type 2
- Data quality and reconciliation
- Data integration
- Structured and unstructured data processing

Programming
- Strong Python
- Strong SQL
- Good understanding of software engineering principles
- Experience developing reusable and production-grade frameworks

Cloud Strong experience in Azure Cloud

Experience with cloud storage, networking, security and cloud data services is preferred.

DevOps

- Git
- CI/CD
- Automated testing
- Databricks deployment automation
- Infrastructure-as-code concepts
- Production monitoring and operational support

Hands-on experience with Palantir Foundry will be strongly preferred because the project involves migration of existing Foundry capabilities.

📌 AT&T Databricks - Lead (Gurugram)
🏢 Prodapt
📍 Gurugram