As a Data Engineer in the Lakehouse and AI Data Platform team, you will design,
build, test and support data pipelines and curated datasets on the firm’s up-to-date
data platform. You will work across ingestion, transformation, modelling,
optimisation and data quality, helping to deliver data products that are
reliable, scalable and fit for purpose. Where there are gaps in platform
functionality, you may also contribute to shared tooling or framework components
that improve how the platform is used and operated.
The role is suited to engineers who are comfortable writing code, working with
SQL and distributed data processing, and solving practical delivery problems in
a team environment. More experienced candidates may also contribute to technical
design, platform standards and the shaping of delivery approaches across a wider
set of use cases.
Key Responsibilities
Pipeline Engineering
* Build, enhance and support batch and streaming data pipelines on the
Lakehouse and AI data platform.
* Refactor or modernise existing data flows where needed to improve
reliability, performance and maintainability.
* Where needed, build reusable tooling to improve delivery, consistency and
operational support.
* Ensure data pipelines are production-ready, well tested and operationally
supportable.
Data Modelling and Curation
* Develop raw, refined and curated datasets that support analytics, reporting
and AI use cases.
* Apply sound data modelling principles to represent business entities,
relationships and historical change accurately.
* Work with consumers to shape data products that are usable, well documented
and aligned to business needs.
Data Quality and Reconciliation
* Implement controls to validate completeness, accuracy and consistency of data
across pipelines and datasets.
* Use reconciliation approaches to build confidence in production outputs and
investigate breaks where they arise.
* Contribute to clear standards for testing, monitoring and issue res