Role Summary NTT DATA is seeking experienced Data Engineers for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg. The engineer will harden, extend and operate reusable migration frameworks developed during the POC, implement Spark/PySpark migration pipelines, integrate Cyber/tokenization services, implement migration controls and automated reconciliation, optimize Iceberg storage and troubleshoot PB-scale migration workloads. This is a highly hands-on engineering role requiring strong distributed-data-processing fundamentals and experience building reliable, restartable and high-performance data pipelines. Experience 10+ years of Data Engineering experience, including 4+ years hands-on production experience with Apache Spark/PySpark. ________________________________________ Key Responsibilities Migration Framework Engineering - Harden and extend a config-driven Python/PySpark migration framework capable of migrating thousands of Hadoop/Hive tables without table-specific code. - Build reusable Hive metadata/schema discovery and migration-configuration components. - Implement migration strategy selection including CTAS/bulk, partition-wise and chunk-wise processing based on table size, partition/file characteristics, tokenization requirements and execution history. - Implement PNC-defined Data Movement Controls including: o checkpointing, o restart/recovery, o idempotency, o retries, o control totals, o audit logging, o exception handling. PB-Scale Migration Execution - Execute Hadoop/Hive → Apache Iceberg historical migrations at multi-TB/PB scale. - Design migration patterns for small, medium, large and very-large tables. - Tune parallel Spark execution across tables, partitions and chunks.



- Implement resumability so failed large-table migrations can restart from safe checkpoints rather than reprocessing entire tables. - Diagnose and optimize Hadoop-read, Spark-processing, tokenization and Iceberg-write bottlenecks. - Support migration-wave execution and production issue resolution. Cyber / Tokenization Integration - Integrate PNC-approved tokenization capabilities, including TaaS/Protegrity where applicable, into migration pipelines. - Implement secure processing of sensitive fields. - Handle service failures, retries, throttling and restart scenarios. - Measure and optimize tokenization impact on end-to-end migration throughput. - Support technical validation of Cyber/tokenization outcomes based on PNC-defined rules. Apache Iceberg Engineering - Create/configure Iceberg tables using approved target standards. - Implement appropriate partitioning and file-sizing strategies. - Optimize compaction and small-file handling. - Work with Iceberg snapshots, schema evolution and metadata. - Troubleshoot Iceberg write/commit/catalog issues. - Support table maintenance and performance optimization. Data Validation & Reconciliation - Develop automated source-target validation covering: o row counts, o schema conformity, o partition completeness, o control totals, o aggregates/checksums, o null/duplicate checks, o tokenization validation where applicable. - Isolate reconciliation failures at table/partition level. - Implement selective retry/reprocessing of failed data. - Produce auditable migration evidence.



Production Engineering - Monitor and troubleshoot distributed Spark jobs. - Analyze Spark UI, executor logs and resource utilization. - Troubleshoot Hadoop/HDFS, Spark, Iceberg catalog, object-storage and tokenization issues. - Build operational telemetry covering migration status, throughput, retries and failures. - Participate in code reviews, testing, CI/CD and engineering standards. - Mentor mid-level/junior Data Engineers. ________________________________________ Mandatory Skills Robust hands-on experience with: - Python - PySpark - Apache Spark / Spark SQL - Hadoop / HDFS - Hive / HiveQL / Hive Metastore - SQL - Apache Iceberg - Parquet - Linux - Git - distributed data processing - large-volume ETL/data migration - source-target reconciliation - Spark performance tuning. Practical Iceberg experience should include: - table creation/read/write, - partitioning, - snapshots, - schema evolution, - file sizing, - compaction, - Spark integration. Strong understanding of: - parallelism, - checkpoint/restart, - idempotency, - retry/recovery, - large-table processing, - small-file problems, - distributed-system troubleshooting. ________________________________________ Preferred Skills - Apache Polaris/Nessie/Glue or another Iceberg catalog. - Kubernetes/OpenShift/Spark Operator. - S3 or S3-compatible object storage; NetApp StorageGRID/ONTAP preferred. - TaaS/Protegrity or equivalent tokenization/masking technology. - PostgreSQL for migration configuration/control/audit metadata. - Trino. - Control-M/Airflow or equivalent enterprise scheduler. - CI/CD for Python/Spark applications. - Banking/financial-services experience. - PB-scale migration experience. - HiveQL→Spark SQL modernization experience. - Bronze/Silver/Gold lakehouse experience.

📌 Data Engineer Advisor (Bengaluru)
🏢 NTT America
📍 Bengaluru