Role Summary NTT DATA is seeking experienced Data Engineers for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg. The engineer will harden, extend and operate reusable migration frameworks developed during the POC, implement Spark/PySpark migration pipelines, integrate Cyber/tokenization services, implement migration controls and automated reconciliation, optimize Iceberg storage and troubleshoot PB-scale migration workloads. This is a highly hands-on engineering role requiring strong distributed-data-processing fundamentals and experience building reliable, restartable and high-performance data pipelines. Experience 10+ years of Data Engineering experience, including 4+ years hands-on production experience with Apache Spark/PySpark. ________________________________________

Key Responsibilities
Migration Framework Engineering
• Harden and extend a config-driven Python/PySpark migration framework capable of migrating thousands of Hadoop/Hive tables without table-specific code.
• Build reusable Hive metadata/schema discovery and migration-configuration components.
• Implement migration strategy selection including CTAS/bulk, partition-wise and chunk-wise processing based on table size, partition/file characteristics, tokenization requirements and execution history.
• Implement PNC-defined Data Movement Controls including: o checkpointing, o restart/recovery, o idempotency, o retries, o control totals, o audit logging, o exception handling. PB-Scale Migration Execution
• Execute Hadoop/Hive → Apache Iceberg historical migrations at multi-TB/PB scale.
• Design migration patterns for small, medium, large and very-large tables.
• Tune parallel Spark execution across tables, partitions and chunks.




• Implement resumability so failed large-table migrations can restart from safe checkpoints rather than reprocessing entire tables.
• Diagnose and optimize Hadoop-read, Spark-processing, tokenization and Iceberg-write bottlenecks.
• Support migration-wave execution and production issue resolution. Cyber / Tokenization Integration
• Integrate PNC-approved tokenization capabilities, including TaaS/Protegrity where applicable, into migration pipelines.
• Implement secure processing of sensitive fields.
• Handle service failures, retries, throttling and restart scenarios.
• Measure and optimize tokenization impact on end-to-end migration throughput.
• Support technical validation of Cyber/tokenization outcomes based on PNC-defined rules. Apache Iceberg Engineering
• Create/configure Iceberg tables using approved target standards.
• Implement appropriate partitioning and file-sizing strategies.
• Optimize compaction and small-file handling.
• Work with Iceberg snapshots, schema evolution and metadata.
• Troubleshoot Iceberg write/commit/catalog issues.
• Support table maintenance and performance optimization. Data Validation & Reconciliation
• Develop automated source-target validation covering: o row counts, o schema conformity, o partition completeness, o control totals, o aggregates/checksums, o null/duplicate checks, o tokenization validation where applicable.
• Isolate reconciliation failures at table/partition level.
• Implement selective retry/reprocessing of failed data.
• Produce auditable migration evidence. Production Engineering
• Monitor and troubleshoot distributed Spark jobs.




• Analyze Spark UI, executor logs and resource utilization.
• Troubleshoot Hadoop/HDFS, Spark, Iceberg catalog, object-storage and tokenization issues.
• Build operational telemetry covering migration status, throughput, retries and failures.
• Participate in code reviews, testing, CI/CD and engineering standards.
• Mentor mid-level/junior Data Engineers. ________________________________________ Mandatory Skills Strong hands-on experience with:
• Python
• PySpark
• Apache Spark / Spark SQL
• Hadoop / HDFS
• Hive / HiveQL / Hive Metastore
• SQL
• Apache Iceberg
• Parquet
• Linux
• Git
• distributed data processing
• large-volume ETL/data migration
• source-target reconciliation
• Spark performance tuning. Practical Iceberg experience should include:
• table creation/read/write,
• partitioning,
• snapshots,
• schema evolution,
• file sizing,
• compaction,
• Spark integration. Robust understanding of:
• parallelism,
• checkpoint/restart,
• idempotency,
• retry/recovery,
• large-table processing,
• small-file problems,
• distributed-system troubleshooting. ________________________________________ Preferred Skills
• Apache Polaris/Nessie/Glue or another Iceberg catalog.
• Kubernetes/OpenShift/Spark Operator.
• S3 or S3-compatible object storage; NetApp StorageGRID/ONTAP preferred.
• TaaS/Protegrity or equivalent tokenization/masking technology.
• PostgreSQL for migration configuration/control/audit metadata.
• Trino.
• Control-M/Airflow or equivalent enterprise scheduler.
• CI/CD for Python/Spark applications.
• Banking/financial-services experience.
• PB-scale migration experience.
• HiveQL→Spark SQL modernization experience.
• Bronze/Silver/Gold lakehouse experience.

📌 Data Engineer Advisor (Bengaluru)
🏢 NTT DATA Services
📍 Bengaluru