We are seeking a highly skilled Palantir Foundry Data Engineer to join our team in developing and implementing end-to-end data solutions. The ideal candidate will have expertise in Palantir Platform, especially Foundry, and be proficient in programming and scripting with Python and PySpark. The role is pivotal in enhancing our data processing capabilities, ensuring data integrity, and driving performance optimization. This position offers an opportunity to work on cutting-edge technologies and contribute to transformative data projects.
Experience Required:
2 to 3 years
Key Responsibilities:
- Implement end-to-end data solutions in Palantir, sourcing data from various platforms.
- Develop and manage ETL pipelines using Foundry Ontology and modular applications.
- Write modular, reusable code for ETL workflows and data transformations.
- Build custom Python functions and libraries to extend Foundry's capabilities.
- Utilize Python libraries like Pandas, NumPy, and PySpark for scalable data operations.
- Optimize performance using parallel computing and performance tuning techniques.
- Ensure data integrity and quality through efficient data processing and automation.
- Collaborate with cross-functional teams to design data models and optimize workflows.
- Handle both structured and unstructured data for ETL pipeline development.
- Implement scalable data processing solutions using Spark (PySpark).
Qualification Criteria:
- Proven experience with Palantir Foundry and its Ontology, pipelines, and modular applications.
- Solid programming and scripting skills in Python and PySpark.
- Ability to clean, transform, and process large datasets efficiently.
- Experience in building and integrating custom Python functions and libraries.
- Proficiency in developing modular, reusable code for ETL workflows.
- Experience with performance optimization using Python libraries like Pandas, NumPy, and PySpark.
- Familiarity with SQL for querying and data modeling.
- Knowledge of building ETL pipelines for data ingestion and transformation.
- Experience in designing data models and optimizing workflows.
- Understanding of big data technologies and distributed systems, particularly Spark (PySpark).
Primary Skills:
- Palantir Foundry
- Python
- PySpark
Key Skills:
- ETL pipelines
- data modeling
- SQL
- data engineering
- distributed systems
- performance optimization
- parallel computing
- data integrity
- cross-functional collaboration
- big data technologies