- Hybrid Data Architecture: Design and implement a seamless data mesh/lakehouse architecture integrating SAP Datasphere (Data & Business Layers and Data Spaces.) with the Databricks Lakehouse Platform.
- Data Pipeline Engineering: Develop and maintain scalable, low-latency ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables (DLT), and Datasphere Replication/Transformation flows.
- Semantic Layer Integration: Expose SAP Datasphere analytical datasets and business models seamlessly to Databricks, ensuring business logic (like hierarchies and currencies) is preserved or effectively translated.
- Federation vs.
Ingestion
Strategy:
Establish governance on when to federate queries live from Databricks to Datasphere versus when to ingest and persist heavy SAP datasets into Delta Lake.
- Unified Governance: Implement end-to-end data governance, lineage tracking and security matrix enforcement utilizing Databricks Unity Catalog alongside Datasphere Spaces and scoping rules.
- Performance Optimization: Monitor and tune data transfer rates, cluster sizes, and query execution plans across both cloud platforms to minimize egress costs and maximize speed.
Must Have Skills:
Technical Core
- SAP Datasphere Expertise: 2 years of production experience with SAP Datasphere, including Space Management, Data Builder, Business Builder, and integration with S/4HANA or BW/4HANA.
- Databricks Mastery: 3+ years of hands-on experience building enterprise-grade lakehouses using Databricks, PySpark, and Delta Lake.
- Advanced Data Ingestion: Deep understanding of modern integration patterns between SAP cloud products and hyperscaler storage (ADLS Gen2, AWS S3) via Kafka, SAP FedML, or OData/CDI streaming.
- Data Governance: Solid experience with Unity Catalog for managing cross-platform data access and data lineage.
- Cross-Ecosystem Fluency: Ability to speak the language of both traditional SAP functional teams (ABAP, S/4 schemas) and modern cloud data science teams.
- Agile & DevOps: Experience with CI/CD tools (Git, Azure DevOps, GitHub Actions) adapted for Databricks notebooks and Datasphere transports.
- Communication: Exceptional communication skills to present architectural decisions to executive stakeholders and technical teams alike.