Job Description: Data Streaming & Platform Engineer

Job Summary

We are seeking an experienced Data Streaming & Platform Engineer to design, deploy, secure, and maintain scalable real-time data streaming platforms using Apache Kafka and Confluent Platform. The ideal candidate will have hands-on experience with distributed systems, Change Data Capture (CDC), stream processing, Kubernetes, infrastructure automation, and production-grade platform operations.

The role involves building low-latency data pipelines, implementing secure and highly available Kafka clusters, supporting cloud and on-premises deployments, and automating platform provisioning and lifecycle management.

Key Responsibilities

1. Kafka & Confluent Platform Engineering

- Design, deploy, configure, and maintain highly available Apache Kafka and Confluent Platform clusters.

- Manage Kafka architecture, including KRaft, brokers, topics, partitions, replication, consumer groups, and cluster sizing.

- Implement Kafka Connect, Schema Registry, Kafka Streams, and related Confluent components.

- Perform Kafka cluster upgrades, migrations, and capacity planning.

- Execute migrations from open-source Kafka to Confluent Platform using MirrorMaker 2, Cluster Linking, and related replication technologies.

- Troubleshoot broker issues, consumer lag, throughput bottlenecks, and production incidents.

2. Real-Time Data Integration & CDC

- Design and implement real-time data pipelines using Debezium CDC connectors for Oracle, MySQL, and Microsoft SQL Server.

- Integrate source databases with Kafka and downstream analytical or operational systems.

- Configure Kafka Connect source and sink connectors, including schema handling and data consistency.

- Implement event-driven architectures for real-time analytics and application integration.

- Troubleshoot connector failures, offset management, duplicate events, and schema evolution issues.

3. Stream Processing

- Develop and maintain stream-processing applications using Apache Flink, Flink SQL, Kafka Streams, and ksqlDB.

- Implement event-time processing, windowing,



aggregations, and real-time metrics computation.

- Optimize stream-processing pipelines for latency, throughput, and reliability.

- Support modernization of batch-based data processing into real-time streaming architectures.

4. Kubernetes & Infrastructure Automation

- Deploy and manage Confluent Platform on Kubernetes using Confluent for Kubernetes (CFK).

- Automate infrastructure provisioning, configuration, deployment, and upgrades using Ansible.

- Develop and maintain Jenkins CI/CD pipelines for platform lifecycle management.

- Work with Docker, Terraform, virtual machines, and on-premises infrastructure.

- Support air-gapped deployments and environments with restricted network access.

- Prepare deployment procedures, technical documentation, and operational runbooks.

5. Security & Access Management

- Configure Kafka security using TLS, mutual TLS (mTLS), SASL, and Kafka ACLs.

- Integrate LDAP and Active Directory for authentication and role-based access control.

- Implement secure access to streaming platforms, monitoring dashboards, and platform services.

- Troubleshoot certificates, authentication failures, authorization issues, and security incidents.

6. Monitoring & Reliability

- Implement monitoring and alerting using Prometheus and Grafana.

- Monitor broker health, Kafka throughput, consumer lag, disk utilization, and cluster performance.

- Perform performance tuning, capacity planning, and root-cause analysis.

- Maintain platform availability and support production deployments and incident resolution.

7. Technical Leadership & Collaboration

- Lead technical proof of concepts (PoCs) and contribute to streaming-platform architecture decisions.





- Collaborate with application, database, DevOps, and analytics teams.

- Mentor engineers on Kafka best practices and distributed streaming concepts.

- Evaluate open-source tools and develop automation utilities for platform sizing and operational efficiency.

Required Technical Skills

- Apache Kafka and Confluent Platform

- Kafka Connect, Schema Registry, KRaft, Kafka Streams, and consumer groups

- Debezium CDC and database integration

- Apache Flink and Flink SQL

- Kubernetes and Confluent for Kubernetes (CFK)

- Ansible, Jenkins CI/CD, Docker, and infrastructure automation

- TLS, mTLS, SASL, LDAP, Active Directory, RBAC, and Kafka ACLs

- Prometheus and Grafana

- Java, Python, and Shell/Bash scripting

- Relational databases, including Oracle, MySQL, Microsoft SQL Server, and PostgreSQL

Preferred Qualifications

- Experience deploying streaming platforms in air-gapped, restricted-network, or regulated environments.

- Experience with on-premises and cloud-based streaming infrastructure.

- Knowledge of Kafka capacity planning, broker sizing, partition strategy, replication, and disaster recovery.

- Experience integrating Kafka with analytical databases.

- Relevant Confluent certifications or equivalent Kafka platform expertise.

Experience & Education

- 5+ years of hands-on experience in Kafka, data streaming, platform engineering, or DevOps.

- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related discipline, subject to employer requirements.

- Relevant industry certifications are an advantage.

Success Measures

- Reliable, highly available streaming-platform deployments.

- Reduced data pipeline latency and improved processing efficiency.

- Automated, repeatable infrastructure provisioning and upgrades.

- Improved platform security, observability, and operational stability.

- Successful Kafka migrations and production implementations.Role & responsibilities

Preferred candidate profile

Perks and advantages

📌 Kafka Developer (Bengaluru)
🏢 Nisource Technologies
📍 Bengaluru