GCP Data Engineer

Level: Manager

Experience: 10–13 Years

Location: Any Metro Location

Notice Period: Immediate to 30 Days / Currently Serving Notice

Compensation: Up to ₹35.5 LPA Fixed

Role Overview

We are looking for an experienced GCP Data Engineer / Data Engineering Manager with 10–13 years of experience in data engineering and strong expertise in the Google Cloud Platform (GCP) ecosystem.

The role involves designing and delivering scalable cloud-based data platforms, optimizing large-scale Spark and Dataproc workloads , managing data orchestration platforms, and providing technical leadership to engineering teams.

The ideal candidate should combine robust hands-on expertise in GCP, Python, Spark, Dataproc, BigQuery and Cloud Composer/Airflow with experience in performance engineering, cost optimization, architecture and stakeholder management.

Key Responsibilities

GCP Data Engineering & Cloud Platform

- Design, develop and implement scalable data engineering solutions on Google Cloud Platform .
- Build data solutions using Cloud Run, Cloud Functions with Python, Cloud SQL PostgreSQL, Pub/Sub, GCS and BigQuery .
- Develop and maintain robust data pipelines and cloud-native data processing solutions.
- Design and manage orchestration workflows using Cloud Composer / Managed Airflow .
- Perform Airflow/Composer optimization, capacity planning and performance tuning.
- Drive FinOps and cloud cost optimization initiatives across data workloads.
- Support Databricks-based data processing and integration .
- Design and integrate REST APIs across cloud applications and data platforms.
- Ensure data platforms are scalable, reliable, secure and cost-efficient.

Performance Engineering & Dataproc Optimization




- Execute workload testing and performance optimization activities for large-scale data workloads.
- Enhance and optimize Python-based execution templates .
- Troubleshoot Spark, YARN and Dataproc failures using Spark UI and platform diagnostics.
- Analyze workload profiles and establish cost-performance benchmarks .
- Define N1/N2 mapping strategies and translate custom machine requirements into standard GCP machine types.
- Recommend appropriate VM sizing, schema design, partitioning and performance improvements .
- Identify bottlenecks across Spark, Dataproc and underlying GCP infrastructure.
- Optimize workloads for both performance and cloud cost efficiency.

Technical Leadership
- Provide technical leadership and architecture guidance to data engineering teams.
- Review technical designs, solutions and implementation approaches.
- Mentor engineers and drive engineering best practices.
- Work closely with business stakeholders, architects and cross-functional teams.
- Own technical delivery and ensure solutions meet performance, scalability and business requirements.
- Participate in solution design, estimation, planning and delivery governance.

Required Technical Skills

Mandatory

- 10–13 years of overall experience in Data Engineering.
- Strong hands-on experience with Google Cloud Platform (GCP) .




- Strong expertise in:
- Python
- Apache Spark
- Dataproc
- Google Cloud Storage (GCS)
- BigQuery
- Cloud Data Architecture
- Experience with:
- Cloud Run
- Cloud Functions
- Cloud SQL – PostgreSQL
- Pub/Sub
- Cloud Composer / Apache Airflow
- Strong understanding of Spark performance tuning and debugging .
- Experience troubleshooting YARN and Dataproc issues .
- Exposure to workload benchmarking, capacity planning and cloud cost optimization.
- Experience with Airflow administration and optimization .
- Strong understanding of data pipeline architecture and large-scale data processing.

Good to Have

- Experience with Databricks .
- Experience designing and consuming REST APIs .
- Knowledge of GCP FinOps / cloud cost governance .
- Experience with custom machine sizing and workload-to-machine-type mapping.
- Experience with enterprise-scale cloud data migration or modernization programs.

Leadership & Behavioral Skills

- Strong stakeholder management and communication skills.
- Ability to lead technical discussions with senior stakeholders.
- Strong analytical and problem-solving abilities.
- Ability to work independently and lead engineering teams.
- Strong solution-design and architecture mindset.
- Ability to balance performance, scalability, reliability and cost .

Candidate Profile We are looking for candidates who have progressed beyond traditional ETL development and possess strong expertise in cloud-native data engineering and performance optimization .

Candidates with strong hands-on experience in GCP + Spark + Dataproc + Python + BigQuery + Cloud Composer/Airflow will be preferred.

📌 GCP (Google Cloud Platform) Data Engineering Manager (10+ yrs, Any Metro city, Immediate Joiners)
🏢 Rbst Consulting
📍 Hyderabad