Contract (part-time hours)

Hours

20–30 hrs/week

Google Cloud Platform, Dataproc, BigQuery, Apache Iceberg, Dataplex, Apache Airflow, Terraform, Apache Spark, Databricks, Data Migration

About the role

You'll replatform Databricks workloads onto GCP-native services, and build the Google Cloud foundations they run on, hands-on. You'll work directly with our product lead, who sets scope and priorities, alongside our Databricks migration engineers.

What you'll do

You own the GCP-native side of the migration, from the landing zone to validated pipelines.

- Build the foundation. Terraform the GCP landing zone: Shared VPC, Private Service Connect, Private Google Access, firewalls, IAM, service accounts and Workload Identity Federation.
- Move the data. Convert Delta tables to Apache Iceberg on GCS, managed through BigLake.
- Map governance. Carry Unity Catalog structures, grants and policies into Dataplex, policy tags and BigQuery row- and column-level controls.
- Replatform pipelines. Convert Spark notebooks and jobs to Dataproc Serverless, orchestration to Cloud Composer, and declarative pipelines to Dataproc with Dataplex data quality.
- Federate. Stand up retain-in-place patterns where Databricks stays and BigLake reads the same Iceberg tables, and move a sample AI workload to Vertex AI.
- Prove parity. Compare source and target on row counts, checksums and performance, and keep runbooks, logs and test results as you go.
- Automate. Check what our agentic engine generates, and script every step you do twice.
- Keep it repeatable. Maintain target environments that deploy and reset cleanly, so runs and demos are repeatable.

Must-havesProduction GCP data platform work is the one non-negotiable; the rest is the toolkit behind it.




- GCP data platforms in production. You've built or migrated production workloads on Dataproc, BigQuery and GCS, not just used BigQuery for analytics.
- Open table formats. Hands-on with Apache Iceberg (BigLake or similar), and enough Delta Lake to map from it.
- Spark depth. PySpark and Spark SQL at scale. You can replace Databricks-specific code, such as dbutils, widgets and %run, with open Spark.
- Governance and orchestration. Dataplex (catalog, policy tags, data quality), and Cloud Composer DAGs you've written yourself.
- Terraform for GCP. Modular, reusable Terraform for networking and IAM, run through CI/CD.
- Databricks literacy. Enough Unity Catalog, jobs and DLT knowledge to read a source estate and plan its GCP-native equivalent.
- Works to a plan. You take direction from a product lead, execute to it, and raise risks early in clear written updates.

Nice to haveAny of these sets a candidate apart.
- Migrated Hadoop, Cloudera or Databricks workloads to Dataproc or BigQuery.
- Delta UniForm, Iceberg REST catalogs or cross-cloud lakehouse federation.
- Vertex AI and agent frameworks.
- Databricks on GCP workspace setup, including customer-managed VPC and PSC.
- Validated LLM-generated code or Terraform against tests.
- Google Cloud Qualified Data Engineer or Professional Cloud Architect certification.

What success looks likeEvery replatform path runs end to end on GCP, with parity evidence behind it.
- Data, governance, pipelines and orchestration each migrate end to end on a representative estate.
- The landing zone deploys cleanly from Terraform in one pass, and reruns without drift.
- Generated code and Terraform are checked before they run, and the misses are logged.
- Target environments reset and rerun reliably for demos.

📌 Senior Databricks Engineer (GCP, AZURE) — Migration & Replatforming (India)
🏢 Settl X
📍 India