Key Responsibilities
- Design and execute Hadoop to Databricks migration strategies for large-scale data platforms.
- Migrate existing HDFS, Hive, Spark, and MapReduce workloads to Databricks.
- Data Engineering – Hadoop to Databricks migration experience
- Solid hands-on experience in migrating and optimizing Spark code
- Analyze existing Hadoop/Spark applications and identify opportunities for modernization and optimization.
- Strong hands-on experience in migrating, refactoring, and optimizing Spark code for Databricks.
- Develop scalable ETL/ELT pipelines using Spark, SQL, and Databricks.
- Optimize Spark jobs by improving partitioning, caching, joins, shuffling, serialization, and memory utilization.
- Work with Databricks Delta Lake / Delta tables for reliable and scalable data processing.
- Implement Delta Lake features such as ACID transactions, schema evolution, time travel, and optimized data layouts.
- Convert legacy Hive SQL / Spark SQL workloads to Databricks SQL and optimize queries.
- Migrate data from HDFS/Hive to cloud-based storage and Databricks.
- Experience with Azure Data Lake Storage (ADLS), AWS S3, or equivalent cloud storage.
- Develop and maintain robust data pipelines using Databricks Workflows and job orchestration tools.
- Implement data quality, validation, reconciliation, and error-handling frameworks during migration.
- Perform source-to-target mapping, data profiling, transformation analysis, and migration validation.
- Troubleshoot performance issues across Spark clusters, notebooks, jobs, and data pipelines.
- Work closely with architects, business analysts, data scientists, and application teams to understand data requirements.
- Implement CI/CD practices for Databricks notebooks, jobs, and code using Git and DevOps tools.
- Follow enterprise security, governance, audit, and compliance standards applicable to banking environments.
- Ensure migrated workloads meet performance, scalability, availability, and data-quality requirements.
📌 Lead BigData Engineer with (Hadoop +Databricks)_ Exp: 7+ Years (India)
🏢 Atyeti
📍 India