Senior Streaming Data Engineer (Confluent / Apache Kafka)

Experience: 3-8 years

Location: [Bangalore]

Employment type: Full-time

About the Role

We are looking for a Senior Streaming Data Engineer to design, operate, and migrate real-time data platforms on Confluent Platform and Confluent Cloud. You will own critical pipelines that move data between applications, databases, and analytics systems, and lead migrations into and out of Confluent.

Experience with other streaming platforms (Kinesis, Pub/Sub, Event Hubs, Pulsar) is valued, as we want someone who can cross-skill quickly and map new platforms onto Kafka.

Responsibilities

- Design and manage Kafka solutions on Confluent: topic design, partitioning strategy, retention, replication, quotas, and ACLs/RBAC
- Lead end-to-end migrations into and out of Confluent (from Kinesis, Pub/Sub, or self-managed Kafka) using MirrorMaker 2, Replicator, Cluster Linking, or custom bridge consumers, including validation and rollback planning
- Guarantee ordering, idempotency, and consumer offset continuity during cutovers
- Build and review producers, consumers, and Kafka Connect pipelines (Debezium CDC included)
- Own schema governance with Schema Registry, including compatibility policies and evolution
- Develop stream processing applications using Flink, Kafka Streams, or ksqlDB
- Design CI/CD and infrastructure-as-code pipelines for consumers, connector configs,



and cluster provisioning
- Monitor cluster health and consumer lag, and tune throughput, latency, and cost
- Implement security: encryption, authentication, and private networking
- Define architecture standards, write runbooks, and mentor junior engineers

Requirements

- 5+ years in data, platform, or distributed systems engineering, including 3+ years with Kafka / Confluent in production
- Deep understanding of Kafka internals and semantics: partitions, consumer groups, rebalancing, offsets, ordering, and exactly-once delivery
- Proven experience with Kafka administration, capacity planning, and troubleshooting production incidents
- Proficiency in Java or Python, strong SQL, and shell scripting
- Strong working knowledge of AWS or GCP (IAM, networking, storage), including VPC, TLS, and private connectivity
- Solid Linux fundamentals: debugging, networking commands, disk and memory behavior
- CI/CD experience (Git with Jenkins, GitHub Actions, or GitLab CI) and monitoring tools (Prometheus, Grafana, Datadog, or CloudWatch)
- Understanding of distributed systems fundamentals: replication, leader election, backpressure, and failure handling

Bonus Skills Hands-on Kinesis or Pub/Sub, Go, Terraform (Confluent provider), Kubernetes (Confluent for Kubernetes, Strimzi), Snowflake/BigQuery, Confluent certification (Developer or Administrator)

📌 Streaming Data Engineer- Bangalore (Bengaluru)
🏢 Celebal Technologies
📍 Bengaluru