Job Description

Insight Global is looking for a Data Engineer to help lead a migration from Glue/Redshift to Databricks

What you will do:

Own the end-to-end migration roadmap: discovery, assessment, design, build, test, cutover, and decommissioning of legacy AWS Glue jobs, Redshift clusters, and associated tooling.

Inventory and prioritize existing pipelines, stored procedures, materialized views, and Redshift workloads; define a wave-based migration plan with transparent success criteria for each wave.

Architect the target Databricks Lakehouse: workspace and account topology, Unity Catalog design, medallion (bronze/silver/gold) layering, storage layout on S3, cluster and SQL Warehouse strategy, and cost guardrails.
- Re-platform pipelines from Glue (PySpark / Spark SQL) and Redshift SQL into Databricks using PySpark, Spark SQL, Delta Live Tables, and/or Databricks Workflows; refactor where it improves performance, maintainability, or cost.
- Define and enforce data parity, reconciliation,



and regression-testing strategy between legacy Redshift and the new Lakehouse during dual-run periods.
- Plan and execute cutover for downstream consumers (BI tools, reverse ETL, ML feature stores, application reads), minimizing disruption.
- Set technical direction and standards
- Establish engineering standards for the new platform: repository structure, CI/CD (e.g., GitHub Actions / Azure DevOps with Databricks Asset Bundles or Terraform), environment promotion, code review, and testing (unit, data quality, integration).
- Define and roll out data quality, observability, and lineage practices (e.g., Great Expectations / DLT expectations, Unity Catalog lineage, monitoring and alerting).
- Govern access, security, and compliance via Unity Catalog: catalogs, schemas, row/column-level security, and audit.

Drive cost optimization: cluster sizing and policies, photon usage, autoscaling, job vs. all-purpose compute decisions, and SQL Warehouse right-sizing.

Required Skills & Experien

📌 Data Engineer (India)
🏢 Insight Global Technologies
📍 India