Role Overview

We are looking for a highly skilled Kafka Migration Engineer to join the ETO Factory team and support a high-visibility on-premises Apache Kafka to AWS MSK migration project.

The candidate will be responsible for Kafka cluster discovery, workload assessment, capacity planning, AWS MSK configuration, migration execution, testing, validation, performance tuning, automation, and legacy Kafka cluster decommissioning.

The ideal candidate should have strong hands-on experience with Apache Kafka, AWS MSK, distributed messaging systems, cloud migration, Linux, programming, automation, and CI/CD, along with the ability to collaborate with application, infrastructure, and platform teams.

Key Responsibilities

- Assess on-premises Kafka clusters and perform discovery across:
- Topics
- Partitions
- Consumer groups
- Throughput patterns
- Retention policies
- Inter-service dependencies
- Evaluate workloads based on:
- Message volume
- Business criticality
- Ordering requirements
- Latency requirements
- Consumer group complexity
- Determine appropriate AWS MSK cluster placement, sizing, and configuration.
- Plan and provision AWS MSK infrastructure, including:
- Broker sizing
- Storage
- Networking
- Security
- Execute UAT and production parallel testing using MirrorMaker 2 or similar replication tools.
- Validate message delivery outcomes and troubleshoot migration discrepancies.




- Perform Kafka/MSK performance tuning and optimization to ensure scalability and production readiness.
- Collaborate with application, infrastructure, and platform teams to finalize:
- Migration plans
- Producer/consumer cutover strategies
- Validation criteria
- Rollback plans
- Execute production cutover activities, including:
- Consumer group migration
- Offset synchronization
- Post-release validation
- Support successful decommissioning of legacy on-premises Kafka clusters.
- Design and implement automation frameworks for repeatable, large-scale Kafka-to-MSK migrations.
- Develop scripts and utilities for:
- Topic creation
- ACL migration
- Consumer group management
- Migration validation
- Explore AI-assisted engineering and automation approaches to improve migration efficiency and quality.
- Document migration procedures, risks, operational learnings, and best practices.

Experience

- 3–5 years of hands-on engineering experience in distributed messaging systems.
- Strong experience working in collaborative, team-based engineering environments.

Programming Strong proficiency in at least one of:
- Python
- Java
- Shell scripting
- Similar programming/scripting languages

Systems & DevOps
- Strong Unix/Linux fundamentals.
- Experience troubleshooting application, deployment, workplace, and runtime issues.
- Understanding of:
- SDLC
- CI/CD
- Change Management
- Kubernetes-based deployments

📌 Kafka Migration Engineer (India)
🏢 Deloitte
📍 India