We are looking for an experienced Senior Data Engineer with strong expertise in Real-Time Streaming, Java, Apache Kafka, Apache Flink, and PySpark to build and operate high-performance, scalable streaming data platforms supporting enterprise Business Platforms and Risk & Compliance initiatives.

The ideal candidate will have extensive experience designing and implementing real-time data pipelines, event-driven architectures, and distributed data processing systems capable of handling high-volume, low-latency data workloads. This role requires strong engineering expertise in up-to-date data platforms, stream processing, and Big Data technologies while collaborating closely with architects, platform teams, and business stakeholders.

Requirements Key Responsibilities

- Design, develop, and maintain scalable real-time data pipelines using Apache Kafka, Apache Flink, Java, and PySpark
- Build high-throughput, low-latency streaming applications supporting enterprise-scale business and regulatory workloads
- Develop event-driven data processing solutions using Kafka and modern streaming technologies
- Design and implement distributed data processing pipelines for real-time analytics and operational reporting
- Collaborate with Data Architects, Platform Engineering teams, Analytics teams, and Business stakeholders to understand requirements and deliver scalable data solutions
- Develop reusable, secure, and maintainable data engineering components following enterprise engineering standards
- Optimize streaming jobs for performance, scalability, fault tolerance, and reliability
- Build and maintain batch and streaming ETL/ELT pipelines using PySpark
- Monitor, troubleshoot, and resolve production issues within streaming data platforms
- Participate in architecture discussions and contribute to the design of modern enterprise data platforms
- Ensure data quality, governance, monitoring, logging, and operational excellence across streaming pipelines
- Participate in Agile/Scrum ceremonies including sprint planning, backlog refinement, stand-ups, and retrospectives




- Create technical documentation, implementation guides, and operational runbooks

Required Technical Skills Programming

- Strong hands-on experience with Java (Mandatory)
- Strong proficiency in Python
- Excellent understanding of Object-Oriented Programming (OOP) principles

Real-Time Streaming

- Apache Kafka
- Apache Flink
- Kafka Streams
- Event-Driven Architecture
- Stream Processing
- Real-Time Data Engineering

Big Data

- PySpark
- Apache Spark
- Distributed Data Processing
- Data Pipeline Development

Databases

- SQL
- Relational Databases
- NoSQL databases (preferred)

Data Engineering

- ETL/ELT Development
- Data Ingestion
- Data Transformation
- Data Integration
- Pipeline Orchestration
- Data Quality
- Performance Optimization

Development & DevOps

- Git
- CI/CD pipelines
- Linux environment
- API Integration
- Monitoring & Logging tools

Required Competencies

- Strong experience designing scalable distributed data systems
- Excellent understanding of real-time streaming architectures
- Ability to build secure, resilient, and fault-tolerant data pipelines
- Strong analytical and problem-solving skills
- Excellent communication and stakeholder management skills
- Ability to work effectively in Agile delivery environments
- Strong ownership mindset with focus on delivery quality and operational excellence
- Ability to collaborate across cross-functional engineering and business teams

Preferred Experience

- Experience working on enterprise-scale data platforms
- Exposure to Risk & Compliance or Business Platform data ecosystems
- Experience supporting mission-critical, high-volume production environments
- Knowledge of cloud-based data engineering platforms is an advantage
- Experience with modern data architecture and distributed systems

Education Bachelor's or Master's Degree in Computer Science, Information Technology, Data Engineering, or a related field (or equivalent industry experience). Boolean Keywords (for better profile quality on job portals) Mandatory Skills:

- Java
- Apache Kafka
- Apache Flink
- PySpark
- Real-Time Streaming
- Data Engineering
- Event-Driven Architecture

📌 Senior Data Engineer - (Real-Time Streaming) (Chennai)
🏢 Global Software Solutions Group
📍 Chennai