We are seeking a highly skilled Data Engineering Architect with deep expertise in Google Cloud Platform (GCP) to architect, design and implement scalable, high-performance data lake solutions. The ideal candidate will have extensive experience in building data ingestion pipelines and managing big data processing projects.
Key Requirements:
Over 12 years of skilled experience in data engineering, specializing in implementing large-scale enterprise Data Engineering projects with the latest technologies.
Over 5 years of hands-on experience in GCP technologies and over 3 years of solution architect experience.
Design and implement end-to-end data architectures leveraging GCP services (e.g., Big Query, Cloud Storage, Dataflow, Data Proc, Pub/Sub, Cloud Composer) for large-scale data ingestion and processing.
Carries extreme hands-on Python coding
Experience in optimising GCP resources for cost and performance
Experience in containerisation of applications
Manage CI/CD using GitHub actions
Exposure to DataPlex, Data Catalog and metadata management
GCP IAM and role management
Valuable to have: exposure of SAP S/4HANA / Salesforce integration with GCP for real-time and batch data processing.
Develop and manage scalable data ingestion pipelines for structured and unstructured data using tools like Cloud Dataflow, Cloud Pub/Sub, and Apache Spark.
Provide architectural guidance for designing secure, scalable, and effective data solutions on the Google Cloud Platform, integrating with on-premise/cloud systems like SAP S/4HANA.
Implement both real-time streaming and batch processing pipelines using Dataflow, and other GCP services to meet business requirements.
Implement data governance, access controls, and security best practices to ensure the integrity, confidentiality, and compliance of data across systems.
Collaborate