**Role Summary** NTT DATA is seeking a highly experienced Senior Data Engineer / Technical Lead for a large-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg. The role requires a strong hands-on engineer who can lead the engineering team while continuing to design, develop, troubleshoot and optimize the solution. The candidate will own technical execution of the migration framework, Spark/PySpark engineering, Iceberg implementation, data modelling/schema mapping, Cyber/TaaS integration, migration controls, reconciliation, performance tuning and technical delivery of migration waves. This is not a coordination-only Technical Lead role. The candidate is expected to contribute to code, conduct design/code reviews, troubleshoot complex production issues and mentor Data Engineers. **Experience** 10+ years of Data Engineering experience, including: - 5+ years hands-on Apache Spark/PySpark. - Strong Hadoop/Hive experience. - Experience leading Data Engineering teams. - Large-scale data migration/data modernization experience. - Solid data modelling and source-to-target mapping experience. Multi-hundred-TB/PB-scale migration experience is strongly preferred. **Key Responsibilities** **Technical Leadership** - Lead a team of Data Engineers through assessment, development, migration, validation and production execution. - Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns. - Own technical estimates, engineering backlog, work allocation and technical delivery. - Conduct design and code reviews and enforce engineering standards. - Mentor Data Engineers and provide hands-on technical guidance. - Act as the primary engineering escalation point for complex migration and production issues. - Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams. - Lead root-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one-off solutions. **Hadoop-to-Iceberg Migration** - Lead technical execution of approximately 2 PB of Hadoop/Hive historical-data migration to Apache Iceberg. - Harden and extend the reusable, configuration-driven migration framework developed during the POC. - Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity. - Implement migration strategies including bulk/CTAS, partition-wise and chunk-wise processing. - Design execution approaches for small, medium, large and very-large tables.



- Optimize parallel migration across tables and partitions. - Implement checkpointing, restart/recovery, idempotency and safe retries. - Lead technical execution and troubleshooting of migration waves. **Data Modelling & Schema Engineering** - Analyze existing Hive data models and translate them into approved Apache Iceberg physical models. - Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration. - Perform source-to-target data mapping and schema/data-type mapping. - Define and implement table structures consistent with approved Iceberg standards. - Analyze: o entities and relationships, o primary/foreign-key relationships, o data types, o partition keys, o business keys, o reference/cross-reference structures. - Identify schema incompatibilities and recommend remediation. - Work with the Data Architect on partitioning, file-layout and model-design decisions. - Ensure migrated models preserve required business relationships and downstream compatibility. - Support future Bronze/Silver/Gold modelling requirements where applicable. Important: enterprise modelling standards remain owned by the Data Architect; the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards. **Spark/PySpark Engineering & Performance** - Develop and review production-grade Python/PySpark components. - Tune Spark jobs for PB-scale migration. - Diagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks. - Optimize table/partition/chunk parallelism. - Tune workloads to meet migration-throughput objectives. - Identify bottlenecks across: Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation - Coordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework. **Apache Iceberg Engineering** - Implement approved Iceberg table standards. - Develop/configure Iceberg tables and Spark→Iceberg processing. - Implement: o partitioning, o schema evolution, o snapshots, o file sizing, o compaction, o metadata maintenance, o small-file remediation. - Troubleshoot Iceberg catalog/write/commit issues.



- Optimize Iceberg table performance and storage characteristics. **Cyber-3 / Tokenization** - Implement PNC-approved TaaS/Protegrity tokenization and retokenization patterns. - Integrate tokenization into Spark migration pipelines. - Implement batching, retry, timeout and recovery handling. - Troubleshoot tokenization failures/performance bottlenecks. - Measure impact of tokenization on end-to-end migration throughput. - Support Cyber/tokenization validation based on PNC-defined rules. **Migration Controls & Reconciliation** - Lead implementation of PNC Data Movement Controls, including: o checkpointing, o idempotency, o control totals, o restart/recovery, o retry, o audit logging, o exception handling. - Build reusable source-to-target reconciliation capabilities covering: o row counts, o schema validation, o partition completeness, o control totals, o aggregates, o hash/checksum where appropriate, o null/duplicate validation, o tokenization validation. - Isolate discrepancies at table/partition level. - Enable selective reprocessing rather than full-table reruns. - Produce technical evidence required for migration acceptance. **Mandatory Skills** Strong hands-on expertise in: - Python - PySpark - Apache Spark / Spark SQL - Hadoop / HDFS - Hive / HiveQL / Hive Metastore - SQL - Apache Iceberg - Data modelling - Source-to-target mapping - Parquet - Linux - Git - Large-volume ETL/data migration - Data validation/reconciliation - Spark performance tuning - Distributed-system troubleshooting. **Data Modelling** Strong experience with: - logical and physical data modelling, - source-to-target mapping, - schema/data-type mapping, - entity relationships, - keys and referential relationships, - dimensional/relational modelling concepts, - large-scale analytical data models. **Technical Leadership** Demonstrated experience: - leading Data Engineering teams, - technical estimation/planning, - detailed solution design, - code/design reviews, - mentoring engineers, - production troubleshooting, - root-cause analysis, - communicating technical issues with architects/customer teams. **Preferred Skills** - Apache Polaris/Nessie/Glue. - OpenShift/Kubernetes/Spark Operator. - S3-compatible object storage. - NetApp StorageGRID/ONTAP. - TaaS/Protegrity or equivalent tokenization technology. - Trino. - PostgreSQL. - Control-M/Airflow. - CI/CD for Spark/Python applications. - Banking/financial-services experience. - PB-scale data migration. - Bronze/Silver/Gold modelling and implementation. - HiveQL→Spark SQL modernization.

📌 Senior Data Engineer (Bengaluru)
🏢 NTT America
📍 Bengaluru