Key Responsibilities
Design, develop, and maintain scalable data platforms and ETL solutions on AWS.
Build and optimize high-performance data pipelines using Python and Spark.
Develop data ingestion, transformation, and processing frameworks leveraging AWS services.
Implement and maintain Lakehouse architecture solutions using contemporary data engineering principles.
Lead migration of existing data processing workloads to Lakehouse platforms utilizing Apache Iceberg capabilities.
Collaborate with architects, product owners, and development teams to design and deliver enterprise data solutions.
Break down solution designs into epics, stories, and technical deliverables.
Ensure data quality, reliability, scalability, and performance across data platforms.
Conduct code reviews, promote engineering best practices, and support quality assurance initiatives.
Create and maintain technical documentation, architecture diagrams, and implementation standards.
Support Agile delivery methodologies and contribute to sprint planning and technical estimations.
Required Skills
Proven experience as a Senior Data Engineer.
Strong expertise in Python and Apache Spark.
Extensive experience building ETL pipelines and data processing frameworks.
Hands-on experience with AWS services including:
AWS EMR
AWS Glue
AWS Lambda
AWS Step Functions
API Gateway
Amazon Athena
Experience working with Data Lakehouse architectures.
Robust understanding of data modelling and enterprise data platform design.
Experience handling large-scale data processing workloads.
Knowledge of data engineering best practices including testing, validation, and performance optimization.
Solid communication and stakeholder management skills.
Preferred Skills
Financial Services or Capital Markets experience.
Experience working with market data and reference data platforms.
Knowledge of Data Governance and Data Management frameworks.
Experience with:
Metadata Manag