We are looking for a Databricks Engineer to join a data modernization program focused on building current data capabilities on Databricks and migrating existing data and application features from Palantir Foundry to the Databricks platform.
The candidate will work closely with data architects, application teams, business stakeholders, and AI/ML engineers to understand existing Palantir functionality, design the equivalent or improved capabilities in Databricks, and develop scalable, production-ready data pipelines and solutions.
The role requires strong hands-on engineering skills in Databricks, Apache Spark, Python, SQL and Delta Lake, along with the ability to understand and translate existing Palantir Foundry implementations into Databricks-native solutions.
Responsibilities
1. Databricks Development
• Design and develop scalable data engineering solutions using Databricks.
• Build batch and near-real-time data ingestion and transformation pipelines.
• Develop data processing workflows using PySpark, Spark SQL and Python.
• Implement data models using Delta Lake and follow Medallion Architecture principles.
• Develop reusable and optimized data pipelines for large datasets.
• Implement data quality, validation, error handling and reconciliation processes.
• Optimize Spark jobs, SQL queries, Delta tables and cluster configurations for performance and cost.
2. Palantir Foundry to Databricks Migration
• Analyze existing Palantir Foundry pipelines, datasets, transformations and business logic.
• Map existing Foundry capabilities to appropriate Databricks technologies and patterns.
• Re-engineer Foundry data pipelines and transformations using Databricks, PySpark, SQL and Delta Lake.
• Identify opportunities to simplify, modernize and improve existing functionality rather than performing a one-to-one migration.
• Perform source-to-target data mapping and migration validation.
• Compare outputs between Palantir and Databricks to ensure functional and data accu