We are seeking a highly skilled Data Engineering Architect with deep expertise in Google Cloud Platform (GCP) to architect, design and implement scalable, high-performance data lake solutions. The ideal candidate will have extensive experience in building data ingestion pipelines and managing big data processing projects.
Key Requirements:
- Over 12 years of professional experience in data engineering, specializing in implementing large-scale enterprise Data Engineering projects with the latest technologies.
- Over 5 years of hands-on experience in GCP technologies and over 3 years of solution architect experience.
- Design and implement end-to-end data architectures leveraging GCP services (e.g., Big Query, Cloud Storage, Dataflow, Data Proc, Pub/Sub, Cloud Composer) for large-scale data ingestion and processing.
- Carries extreme hands-on Python coding
- Experience in optimising GCP resources for cost and performance
- Experience in containerisation of applications
- Manage CI/CD using GitHub actions
- Exposure to DataPlex, Data Catalog and metadata management
- GCP IAM and role management
- Good to have: exposure of SAP S/4HANA / Salesforce integration with GCP for real-time and batch data processing.
- Develop and manage scalable data ingestion pipelines for structured and unstructured data using tools like Cloud Dataflow, Cloud Pub/Sub, and Apache Spark.
- Provide architectural guidance for designing secure, scalable, and effective data solutions on the Google Cloud Platform, integrating with on-premise/cloud systems like SAP S/4HANA.
- Implement both real-time streaming and batch processing pipelines using Dataflow, and other GCP services to meet business requirements.
- Implement data governance, access controls, and security best practices to ensure the integrity, confidentiality, and compliance of data across systems.
- Collaborate