Key Responsibilities

- Design and execute Hadoop to Databricks migration strategies for large-scale data platforms.

- Migrate existing HDFS, Hive, Spark, and MapReduce workloads to Databricks.

- Data Engineering – Hadoop to Databricks migration experience

- Strong hands-on experience in migrating and optimizing Spark code

- Analyze existing Hadoop/Spark applications and identify opportunities for modernization and optimization.

- Robust hands-on experience in migrating, refactoring, and optimizing Spark code for Databricks.

- Develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks.

- Optimize Spark jobs by improving partitioning, caching, joins, shuffling, serialization, and memory utilization.

- Work with Databricks Delta Lake / Delta tables for reliable and scalable data processing.

- Implement Delta Lake features such as ACID transactions, schema evolution, time travel, and optimized data layouts.

- Convert legacy Hive SQL / Spark SQL workloads to Databricks SQL and optimize queries.

- Migrate data from HDFS/Hive to cloud-based storage and Databricks.





- Experience with Azure Data Lake Storage (ADLS), AWS S3, or equivalent cloud storage.

- Develop and maintain robust data pipelines using Databricks Workflows and job orchestration tools.

- Implement data quality, validation, reconciliation, and error-handling frameworks during migration.

- Perform source-to-target mapping, data profiling, transformation analysis, and migration validation.

- Troubleshoot performance issues across Spark clusters, notebooks, jobs, and data pipelines.

- Work closely with architects, business analysts, data scientists, and application teams to understand data requirements.

- Implement CI/CD practices for Databricks notebooks, jobs, and code using Git and DevOps tools.

- Follow enterprise security, governance, audit, and compliance standards applicable to banking environments.

- Ensure migrated workloads meet performance, scalability, availability, and data-quality requirements.

📌 Lead BigData Engineer with (Hadoop +Databricks)_ Exp: 7+ Years (India)
🏢 Atyeti
📍 India