Job Summary

We are seeking a highly skilled Senior Data Engineer with deep expertise in Testing, validating, Developing and supporting large-scale batch and real-time data platforms built on Apache Spark, Apache Kafka, and Apache NiFi. The ideal candidate will have a solid background in distributed systems, streaming architectures, event-driven platforms, and cloud-native data processing. The successful candidate will collaborate with cross-functional teams to design innovative solutions, enhance platform capabilities, and ensure operational excellence in production environments.
Key Responsibilities
- Develop, test, validate and maintain, scalable batch and real-time data processing applications using Apache Spark, Kafka, and NiFi.
- Build validation suite for high-performance, fault-tolerant, and resilient distributed systems capable of handling large data volumes.
- Develop reusable frameworks, libraries, and shared platform components to accelerate engineering productivity.
- Translate business and technical requirements into scalable software solutions.




- Contribute to architecture discussions, technical designs, and engineering standards.
Streaming & Data Platform Engineering
- Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently.
- Validate and support Apache NiFi data ingestion, transformation, and routing workflows.
- Build reliable data pipelines supporting real-time and near-real-time processing requirements.
- Implement solutions for data replay, recovery, checkpoint management, and failure handling.
Performance & Scalability
- Analyze system bottlenecks and optimize application performance, throughput, and resource utilization.
- Improve scalability, reliability, and availability of distributed applications.
- Develop and test solutions leveraging modern storage technologies including Apache Ozone, Ceph, and cloud-native storage platforms.
- Build deployment automation and operati

📌 Sr Data Engineer (Pune)
🏢 Mastercard
📍 Pune