Databricks Data Engineer (GCP BigQuery to Azure Databricks Migration)

Location: Bangalore / Hyderabad / Pune
Experience: 6-12 Years

We are looking for a highly skilled Databricks Data Engineer to join a large-scale enterprise cloud modernization program focused on migrating data platforms from Google Cloud Platform (BigQuery/GDAP) to Azure Databricks Lakehouse.

The ideal candidate will have hands-on experience in BigQuery migration projects, modern data engineering practices, Azure Databricks, PySpark, Delta Lake, and large-scale ETL modernization initiatives.

This role requires strong technical expertise in designing scalable data pipelines, optimizing lakehouse architectures, and ensuring successful end-to-end migration of enterprise data assets.

Key Responsibilities

- Design, develop, and optimize scalable ETL/ELT pipelines using Azure Databricks and PySpark.
- Migrate BigQuery datasets, tables, views, and schemas to Azure Databricks Delta Lake.
- Implement Medallion Architecture (Bronze, Silver, Gold) and Lakehouse best practices.
- Develop robust batch and incremental data processing frameworks.
- Build data validation, reconciliation, and quality assurance frameworks.
- Perform SQL optimization, Spark performance tuning, and workload optimization.
- Support historical data migration, testing, deployment, and production stabilization.
- Collaborate with architects, business stakeholders, and data platform teams during the migration journey.




- Implement CI/CD processes and version control using Git-based repositories.
- Ensure data governance standards through Unity Catalog and enterprise security controls.

Mandatory Skills

Core Databricks & Spark

- Azure Databricks
- PySpark
- Spark SQL
- Delta Lake
- Unity Catalog
- Data Lakehouse Architecture
- Medallion Architecture

Azure Data Engineering

- Azure Data Lake Storage (ADLS Gen2)
- Azure Data Factory (ADF) (Preferred)
- Data Pipeline Development
- ETL/ELT Frameworks

Migration Expertise (Must Have)

- Google BigQuery
- GCP to Azure Migration
- BigQuery Schema Conversion
- Data Migration Strategy
- Data Validation & Reconciliation Frameworks

Development & DevOps

- SQL Performance Tuning
- Git
- CI/CD Pipelines
- Production Deployment & Support

Preferred Skills

- Airflow / Cloud Composer
- Google Dataflow
- DBT (Data Build Tool)
- Azure Key Vault
- Terraform
- Data Governance & Metadata Management
- Agile/Scrum Delivery Model

Qualifications

- 6-12 years of Data Engineering experience.
- Minimum 3+ years of hands-on Azure Databricks experience.
- Proven experience in at least one large-scale cloud migration program involving BigQuery.
- Strong understanding of contemporary cloud-native data architectures and Lakehouse concepts.

Looking for short Notice joiners, if interested, Pls share your resume to [email protected]

Total Exp -

Current Location -

Current CTC -

Expected CTC -

Notice period -

Current Company -

📌 Databricks Data Engineer (GCP BigQuery To Azure Databricks Migration (Hyderabad)
🏢 Tech Mahindra
📍 Hyderabad