Role Overview
We are building a bespoke data platform focused on Financial Crime transaction analysis. The platform processes tens of millions of transaction records daily and enables large-scale analytics to identify suspicious patterns and behavioral anomalies.
We are looking for a hands-on Data Engineer with strong Python and Spark expertise who can design scalable data pipelines, perform large-scale data analysis, and contribute to the architecture of a high-volume data platform.
This role is ideal for engineers who enjoy deep coding, distributed data processing, and building custom data platforms rather than relying purely on managed tools.
Key Responsibilities
Build and maintain scalable data pipelines across stages such as data download, ingestion, and analysis within a bespoke data platform.
Develop high-performance data processing logic using Python and PySpark.
Perform large-scale transaction data analysis using custom algorithms and in-house logic to detect financial crime patterns.
Work with high-volume datasets (80–90 million records processed regularly)
and optimize pipeline performance.
Design efficient data models and transformations for large-scale processing.
Write optimized SQL queries on PostgreSQL (RDS), leveraging:
- Window functions
- Partitioning
- Query performance optimization
Work with Delta tables and Parquet-based storage formats for efficient data processing.
Build and maintain Spark batch and streaming jobs.
Implement engineering best practices including unit testing, static code analysis, and CI/CD practices.
Contribute to data platform architecture and system design decisions.