Key Responsibilities
- Design, develop, and maintain scalable data platforms and ETL solutions on AWS.
- Build and optimize high-performance data pipelines using Python and Spark.
- Develop data ingestion, transformation, and processing frameworks leveraging AWS services.
- Implement and maintain Lakehouse architecture solutions using modern data engineering principles.
- Lead migration of existing data processing workloads to Lakehouse platforms utilizing Apache Iceberg capabilities.
- Collaborate with architects, product owners, and development teams to design and deliver enterprise data solutions.
- Break down solution designs into epics, stories, and technical deliverables.
- Ensure data quality, reliability, scalability, and performance across data platforms.
- Conduct code reviews, promote engineering best practices, and support quality assurance initiatives.
- Create and maintain technical documentation, architecture diagrams, and implementation standards.
- Support Agile delivery methodologies and contribute to sprint planning and technical estimations.
Required Skills
- Proven experience as a Senior Data Engineer.
- Strong expertise in Python and Apache Spark.
- Extensive experience building ETL pipelines and data processing frameworks.
- Hands-on experience with AWS services including:
- AWS EMR
- AWS Glue
- AWS Lambda
- AWS Step Functions
- API Gateway
- Amazon Athena
- Experience working with Data Lakehouse architectures.
- Solid understanding of data modelling and enterprise data platform design.
- Experience handling large-scale data processing workloads.
- Knowledge of data engineering best practices including testing, validation, and performance optimization.
- Strong communication and stakeholder management skills.
Preferred Skills
- Financial Services or Capital Markets experience.
- Experience working with market data and reference data platforms.
- Knowledge of Data Governance and Data Management frameworks.
- Experience with:
- Metadata Manag