Job Summary

We are seeking a highly skilled Senior Data Engineer with deep expertise in Testing, validating, Developing and supporting large-scale batch and real-time data platforms built on Apache Spark, Apache Kafka, and Apache NiFi. The ideal candidate will have a solid background in distributed systems, streaming architectures, event-driven platforms, and cloud-native data processing. The successful candidate will collaborate with cross-functional teams to design innovative solutions, enhance platform capabilities, and ensure operational excellence in production settings.
Key Responsibilities
Develop, test, validate and maintain, scalable batch and real-time data processing applications using Apache Spark, Kafka, and NiFi.
Build validation suite for high-performance, fault-tolerant, and resilient distributed systems capable of handling large data volumes.
Develop reusable frameworks, libraries, and shared platform components to accelerate engineering productivity.
Translate business and technical requirements into scalable software solutions.




Contribute to architecture discussions, technical designs, and engineering standards.
Streaming & Data Platform Engineering
Create and implement validation suite for Spark batch and streaming applications to process high-volume datasets efficiently.
Validate and support Apache NiFi data ingestion, transformation, and routing workflows.
Build reliable data pipelines supporting real-time and near-real-time processing requirements.
Implement solutions for data replay, recovery, checkpoint management, and failure handling.
Performance & Scalability
Analyze system bottlenecks and optimize application performance, throughput, and resource utilization.
Improve scalability, reliability, and availability of distributed applications.
Develop and test solutions leveraging up-to-date storage technologies including Apache Ozone, Ceph, and cloud-native storage platforms.
Build deployment automation and operati

📌 Sr Data Engineer Pune
🏢 Mastercard
📍 Pune