The Opportunity

To design and implement real-time data streaming pipelines ensuring 24 7 availability of high-volume transactional data. Your role includes building fault-tolerant, scalable systems using modern big data technologies, integrating data from multiple sources into data lakes/lakehouses, and enabling downstream analytics. You will collaborate with internal teams (Data Science, Product, and Technology) to deliver reliable data solutions, optimize performance, and maintain compliance with NPCI standards.

Job Details
- Job Title: Data Engineer (Streaming)
- Division: Data Analytics
- Years of Experience: 38 years
- Education: Graduation in Computer Science/IT (preferably BE/B.Tech) or equivalent; advanced degrees are a plus
- Employment Type: Full time, Permanent
- Location: Mumbai Hyderabad

Key Responsibilities
- Design and develop real-time data pipelines using Apache Kafka and stream processing frameworks (Spark Structured Streaming / Apache Flink).
- Ensure 24 7 data availability with fault-tolerant, highly reliable systems.
- Implement ingestion, transformation,



and load (ITL) patterns for data lakes/lakehouses (S3/MinIO, HDFS).
- Work with table formats like Iceberg, Hudi, or Paimon for ACID transactions and schema evolution.
- Optimize SQL queries on Trino/Hive for large-scale analytics.
- Develop orchestration workflows using DBT, Dagster, or Airflow for data transformations.
- Write efficient code in Python, Scala, and Java for data processing and automation.
- Collaborate with cross-functional teams to understand requirements and deliver high-quality data solutions.
- Monitor pipeline health, manage checkpoints, and implement observability for streaming jobs.
- Ensure compliance with security, governance, and audit standards.

Requirements

Key Skills and Experience Required
- Mandatory Technical Skills:
- Apache Kafka (topics, partitions, offsets, reliability)
- Stream processing (Spark Structured Streaming or Apache Flink)
- Preferred Technic

📌 Senior Associate Data Engineering (Hyderabad)
🏢 National Payments Corporation Of India (NPCI)
📍 Hyderabad