GCP Engineering Skills
Google Cloud Platform (GCP)
BigQuery
Dataproc / Apache Spark
Bigtable
Dataplex / Data Catalog
Python (ETL/ELT pipelines and automation)
SQL (data transformation, validation, reconciliation)
Databases:
MySQL
Hive
MongoDB
Cloud Composer / Apache Airflow
Kafka and Google Pub/Sub
Apigee / REST APIs (data extraction and integration)
GenAI (LLMs, embeddings, prompt engineering) and integration with data platforms
Experience with Vertex AI, BigQuery ML, or GenAI-enabled GCP services (preferred)
Key Responsibilities
Migration & Modernization
Design and execute comprehensive migration strategies from on-premises, legacy platforms, or other cloud settings to GCP.
Migrate structured and semi-structured data to BigQuery, Bigtable, and other GCP storage solutions.
Build, optimize, and maintain scalable migration pipelines using Python, SQL, and Dataproc (Apache Spark).
Implement batch and streaming migration patterns using:
Cloud Composer (Apache Airflow)
Kafka
Google Pub/Sub
Data Quality & Governance
Perform schema mapping, data transformation, reconciliation, and validation to ensure accuracy, consistency, and completeness of migrated datasets.
Enable metadata management, lineage, and governance using Dataplex and Data Catalog.
Handle large-volume data transfers using best practices including:
Parallel loading
Incremental loads
CDC (Change Data Capture)
GenAI Adoption
Leverage GenAI and automation techniques to accelerate:
Data discovery
Schema understanding
Mapping recommendations
Migration assessments
Apply GenAI-assisted:
Data transformation
Code generation
Migration pattern optimization
SQL conversion
Pipeline scaffolding
Test case generation