Role summary
We are looking for an Expert Software Engineer to design, build, and scale our next-generation Data Platform and Data-Driven APIs. This role combines distributed data processing (Apache Spark) with platform and microservices engineering (Java) to enable reliable, scalable, and real-time data access.
You will operate at the intersection of data engineering and backend platform engineering-building systems that not only process large volumes of data but also expose that data through robust, well-designed APIs and services.
This role goes beyond implementing requirements. We expect engineers to understand business context, challenge assumptions, and take end-to-end ownership of delivering meaningful outcomes.
Key responsibilities
Data Platform Engineering
Design and develop scalable data pipelines using Apache Spark (batch and streaming)
Build and maintain data platform layers: ingestion, transformation, and serving
Optimize Spark jobs for performance, cost, and reliability (partitioning, skew handling, memory tuning)
Implement data quality, observability,
and lineage frameworks
Contribute to data architecture decisions (Lakehouse, data mesh, storage formats, partition strategies)
Define and enforce data contracts and schema evolution practices
Platform APIs & Backend Engineering
Design and build data-driven platform APIs using Java (preferred)
Develop microservices that expose curated datasets for product and partner consumption
Implement RESTful APIs and event-driven services for real-time and near real-time data access
Ensure low-latency, high-availability data serving layers
Integrate with upstream/downstream systems, including legacy APIs where required
Cloud & Platform Integration
Build and deploy solutions on Azure (preferred) / AWS / GCP
Leverage cloud-native services for data storage, compute, and messaging
Work with event streaming systems (Kafka/Event Hubs) for real-time pipelines
Support containerized deployments and orchestration (Kubernete