- Design, develop, and maintain scalable data platforms and ETL solutions on AWS.
- Build and optimize high-performance data pipelines using Python and Spark.
- Develop data ingestion, transformation, and processing frameworks leveraging AWS services.
- Implement and maintain Lakehouse architecture solutions using modern data engineering principles.
- Lead migration of existing data processing workloads to Lakehouse platforms utilizing Apache Iceberg capabilities.
- Collaborate with architects, product owners, and development teams to design and deliver enterprise data solutions.
- Break down solution designs into epics, stories, and technical deliverables.
- Ensure data quality, reliability, scalability, and performance across data platforms.
- Conduct code reviews, promote engineering best practices, and support quality assurance initiatives.
- Create and maintain technical documentation, architecture diagrams, and implementation standards.
- Support Agile delivery methodologies and contribute to sprint planning and technical estimations.
Required Skills
- Proven experience as a Senior Data Engineer.
- Robust expertise in Python and Apache Spark.
- Extensive experience building ETL pipelines and data processing frameworks.
- Hands-on experience with AWS services including:
- AWS EMR
- AWS Glue
- AWS Lambda
- AWS Step Functions
- API Gateway
- Amazon Athena
- Experience working with Data Lakehouse architectures.
- Strong understanding of data modelling and enterprise data platform design.
- Experience handling large-scale data processing workloads.
- Knowledge of data engineering best practices including testing, validation, and performance optimization.
- Strong communication and stakeholder management skills.
Preferred Skills
- Financial Services or Capital Markets experience.
- Experience working with market data and reference data platforms.
- Knowledge of Data Governance and Data Management frameworks.
- Experience with:
- Metadata Management
- Master Data Management (MDM)
- Data Quality frameworks
- AWS Certifications such as:
- AWS Certified Solutions Architect
- AWS Certified Data Analytics
Bonus Skills
- Knowledge of Indices and Index Construction methodologies.
- Asset Management domain experience.
- Exposure to modern open-table formats such as Apache Iceberg.
Desired Candidate Profile
- Strong technical leadership and mentoring capabilities.
- Excellent problem-solving and analytical skills.
- Ability to communicate complex technical concepts to technical and non-technical audiences.
- Experience working in Agile product teams.
- Strong ownership mindset with a focus on delivering scalable and maintainable solutions.
- Passion for modern cloud-based data engineering and platform modernization.