Job Description: Databricks DB Migration Engineer

Experience:6-8Years

Locations: Pan India

Role Title

Databricks DB Migration Engineer

Experience

6-8 years; 4+ years in data/database migration and 4+ years hands-on with Databricks preferred

Engagement Type

Full-time / Project-based

Primary Skills

Databricks, Apache Spark, PySpark, SQL, Delta Lake, ETL/ELT, Database Migration

Cloud Platforms

Azure preferred; AWS/GCP exposure added advantage

Reporting To

Data Platform Architect / Cloud Migration Lead / Delivery Manager

Role Overview

We are looking for a Databricks DB Migration Engineer responsible for planning, designing, executing, and validating database migration and modernization initiatives from legacy/on-premises or cloud data platforms to the Databricks Lakehouse platform. The role requires strong hands-on engineering capability in SQL, PySpark, Apache Spark, Delta Lake, data ingestion, performance tuning, reconciliation, and production cutover activities.

Key Responsibilities

- Assess existing source database environments, schemas, data volumes, dependencies, jobs, stored procedures, ETL pipelines, and reporting workloads for migration readiness.
- Design migration approach for moving data and workloads from RDBMS, data warehouse, Hadoop, or cloud databases to Databricks Lakehouse using batch and incremental migration patterns.
- Develop scalable ingestion and transformation pipelines using PySpark, Spark SQL, Databricks Workflows, Delta Live Tables, Auto Loader, and notebooks/jobs as applicable.
- Convert and optimize SQL scripts, stored procedures, ETL logic, and business transformations into Spark SQL/PySpark-based implementations.
- Implement Delta Lake best practices including partitioning, OPTIMIZE, Z-ORDER, schema evolution, time travel, ACID transactions, and data retention strategies.
- Perform data profiling, cleansing, mapping, validation, reconciliation, and row/hash/count-level comparison between source and target systems.
- Tune Spark jobs and Databricks clusters for performance, cost, concurrency, workload isolation, and operational efficiency.
- Support migration dry runs, defect triage, production cutover, rollback planning, post-migration validation, hypercare, and operational handover.
- Collaborate with architects, DBAs, data engineers, cloud teams, security teams, QA teams,



and business stakeholders to ensure successful migration delivery.
- Prepare technical documentation including migration design, mapping documents, runbooks, reconciliation reports, deployment plans, and operational support guides.

Mandatory Skills

- Strong hands-on experience with Databricks workspace, clusters, notebooks, jobs/workflows, Unity Catalog or workspace-level security controls.
- Strong programming experience in PySpark and Spark SQL; Python scripting experience is required.
- Good understanding of database migration lifecycle: assessment, planning, schema conversion, data migration, validation, cutover, and post-production support.
- Experience working with source systems such as Oracle, SQL Server, Teradata, Netezza, PostgreSQL, MySQL, DB2, Hadoop/Hive, Snowflake, Redshift, Synapse, or BigQuery.
- Good knowledge of relational database concepts, SQL tuning, query optimization, indexing concepts, schemas, constraints, and metadata analysis.
- Hands-on exposure to ETL/ELT tools or frameworks such as Azure Data Factory, Informatica, Talend, SSIS, dbt, Airflow, or custom Spark-based pipelines.
- Experience in data validation, reconciliation, data quality checks, and migration test automation.
- Understanding of cloud storage and data formats such as ADLS Gen2/S3/GCS, Parquet, JSON, CSV, Avro, and Delta format.
- Working knowledge of CI/CD, Git, release management, environment promotion, and deployment best practices.

Preferred Skills

- Databricks Certified Data Engineer Associate/Professional certification.
- Experience with Azure Databricks, Azure Data Factory, ADLS Gen2, Azure SQL, Synapse, Event Hubs, Key Vault, and Azure DevOps.
- Experience in migration accelerators, code conversion utilities, metadata-driven migration frameworks, or automated reconciliation frameworks.
- Knowledge of Unity Catalog, data governance, lineage, access control, masking, and compliance requirements.
- Exposure to medallion architecture,



Lakehouse design patterns, CDC, streaming ingestion, and incremental processing.
- Experience in Agile/Scrum delivery model and working with distributed teams.

Required Technical Competencies

- Databricks Lakehouse engineering and Delta Lake implementation.
- Spark performance tuning including partition management, caching, shuffle optimization, file sizing, broadcast joins, and cluster sizing.
- Schema migration, metadata analysis, data type mapping, and SQL-to-Spark conversion.
- Batch migration, incremental migration, CDC-based migration, and large-volume data movement.
- Data quality, validation, exception handling, audit logging, and operational monitoring.
- Production support, job scheduling, failure recovery, alerting, SLA adherence, and incident resolution.

Databricks certification is preferred.

- Cloud certification in Azure/AWS/GCP is an added advantage.

Soft Skills

- Strong analytical and problem-solving skills with attention to data accuracy and migration quality.
- Ability to communicate technical issues clearly to stakeholders, architects, and delivery leadership.
- Robust ownership mindset with ability to work independently in high-pressure migration windows.
- Good documentation discipline and ability to create reusable migration assets and runbooks.
- Ability to collaborate with cross-functional teams across application, database, infrastructure, security, and business groups.

Deliverables / Success Metrics

- Successful migration of source data and workloads to Databricks within agreed timelines and quality thresholds.
- Validated reconciliation reports with documented data quality outcomes.
- Optimized Databricks jobs meeting required SLA, cost, and performance targets.
- Complete migration documentation, runbooks, support handover, and known issue logs.
- Low production defects during cutover and hypercare phase.

Sample Interview Focus Areas

- Explain Delta Lake advantages for migration workloads.
- Approach to migrate stored procedures from Oracle/SQL Server/Teradata to Databricks.
- Design a reconciliation framework for large-volume migration.
- How to tune slow-running Spark SQL/PySpark jobs in Databricks.
- How to plan incremental migration and cutover with minimal downtime.

📌 Data Migration + Databricks (Uttar Pradesh)
🏢 Tata Consultancy Services
📍 Uttar Pradesh