Role Summary

NTT DATA is seeking experienced Data Engineers for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.

The engineer will harden, extend and operate reusable migration frameworks developed during the POC, implement Spark/PySpark migration pipelines, integrate Cyber/tokenization services, implement migration controls and automated reconciliation, optimize Iceberg storage and troubleshoot PB-scale migration workloads.

This is a highly hands-on engineering role requiring strong distributed-data-processing fundamentals and experience building reliable, restartable and high-performance data pipelines.

Experience

10+ years of Data Engineering experience, including 4+ years hands-on production experience with Apache Spark/PySpark.

________________________________________

Key Responsibilities

Migration Framework Engineering

- Harden and extend a config-driven Python/PySpark migration framework capable of migrating thousands of Hadoop/Hive tables without table-specific code.
- Build reusable Hive metadata/schema discovery and migration-configuration components.
- Implement migration strategy selection including CTAS/bulk, partition-wise and chunk-wise processing based on table size, partition/file characteristics, tokenization requirements and execution history.
- Implement PNC-defined Data Movement Controls including:

o checkpointing,

o restart/recovery,

o idempotency,

o retries,

o control totals,

o audit logging,

o exception handling.

PB-Scale Migration Execution

- Execute Hadoop/Hive → Apache Iceberg historical migrations at multi-TB/PB scale.
- Design migration patterns for small, medium, large and very-large tables.
- Tune parallel Spark execution across tables, partitions and chunks.




- Implement resumability so failed large-table migrations can restart from safe checkpoints rather than reprocessing entire tables.
- Diagnose and optimize Hadoop-read, Spark-processing, tokenization and Iceberg-write bottlenecks.
- Support migration-wave execution and production issue resolution.

Cyber / Tokenization Integration

- Integrate PNC-approved tokenization capabilities, including TaaS/Protegrity where applicable, into migration pipelines.
- Implement secure processing of sensitive fields.
- Handle service failures, retries, throttling and restart scenarios.
- Measure and optimize tokenization impact on end-to-end migration throughput.
- Support technical validation of Cyber/tokenization outcomes based on PNC-defined rules.

Apache Iceberg Engineering

- Create/configure Iceberg tables using approved target standards.
- Implement appropriate partitioning and file-sizing strategies.
- Optimize compaction and small-file handling.
- Work with Iceberg snapshots, schema evolution and metadata.
- Troubleshoot Iceberg write/commit/catalog issues.
- Support table maintenance and performance optimization.

Data Validation & Reconciliation
- Develop automated source-target validation covering:

o row counts,

o schema conformity,

o partition completeness,

o control totals,

o aggregates/checksums,

o null/duplicate checks,

o tokenization validation where applicable.

- Isolate reconciliation failures at table/partition level.
- Implement selective retry/reprocessing of failed data.
- Produce auditable migration evidence.





Production Engineering

- Monitor and troubleshoot distributed Spark jobs.
- Analyze Spark UI, executor logs and resource utilization.
- Troubleshoot Hadoop/HDFS, Spark, Iceberg catalog, object-storage and tokenization issues.
- Build operational telemetry covering migration status, throughput, retries and failures.
- Participate in code reviews, testing, CI/CD and engineering standards.
- Mentor mid-level/junior Data Engineers.

________________________________________

Mandatory Skills

Strong hands-on experience with:
- Python
- PySpark
- Apache Spark / Spark SQL
- Hadoop / HDFS
- Hive / HiveQL / Hive Metastore
- SQL
- Apache Iceberg
- Parquet
- Linux
- Git
- distributed data processing
- large-volume ETL/data migration
- source-target reconciliation
- Spark performance tuning.

Practical Iceberg experience should include:
- table creation/read/write,
- partitioning,
- snapshots,
- schema evolution,
- file sizing,
- compaction,
- Spark integration.

Robust understanding of

- parallelism,
- checkpoint/restart,
- idempotency,
- retry/recovery,
- large-table processing,
- small-file problems,
- distributed-system troubleshooting.

________________________________________

Preferred Skills

- Apache Polaris/Nessie/Glue or another Iceberg catalog.
- Kubernetes/OpenShift/Spark Operator.
- S3 or S3-compatible object storage; NetApp StorageGRID/ONTAP preferred.
- TaaS/Protegrity or equivalent tokenization/masking technology.
- PostgreSQL for migration configuration/control/audit metadata.
- Trino.
- Control-M/Airflow or equivalent enterprise scheduler.
- CI/CD for Python/Spark applications.
- Banking/financial-services experience.
- PB-scale migration experience.
- HiveQL→Spark SQL modernization experience.
- Bronze/Silver/Gold lakehouse experience.

📌 Data Engineer Advisor (Bengaluru)
🏢 NTT DATA Services
📍 Bengaluru