Job Description

Key Responsibilities

n

▪

n

Assess on-prem Kafka clusters and perform discovery across topics, partitions, consumer groups, throughput patterns, retention policies, and inter-service dependencies to determine migration readiness.

n

▪

n

Analyze workloads based on message volume, business criticality, ordering guarantees, latency requirements, and consumer group complexity to determine appropriate MSK cluster placement and configuration.

n

▪

n

Plan and provision required AWS MSK infrastructure capacity including broker sizing, storage, networking, and security configurations to support migration, testing, and production execution.

n

▪

n

Execute UAT and production parallel testing using MirrorMaker 2 or similar replication tooling, compare message delivery outcomes, capture evidence, and troubleshoot discrepancies through closure.

n

▪

n

Perform performance tuning and optimization of MSK clusters to ensure migrated workloads are stable, scalable, efficient, and production ready.

n

▪

n

Partner with application engineering, infrastructure, and platform teams to finalize migration plans, consumer/producer cutover approach, validation criteria, and rollback considerations.

n

▪

n

Manage release readiness, execute production cutover activities including consumer group migration, offset synchronization, and complete post-release checkout and validation procedures.

n

▪

n

Support legacy on-premises Kafka cluster decommissioning after successful migration and validation.

n

▪

n

Design, enhance, and implement automation frameworks and AI-assisted solutions to enable repeatable, efficient, large-scale Kafka-to-MSK migrations.

n

▪

n

Document migration procedures, operational learnings, risks, and best practices to improve the factory execution model.

n

Required Qualifications

n

Basic Qualifications

n

▪

n

Education: Bachelor's or Master's degree in Computer Science, Engineering, Applied Mathematics,



or a related quantitative discipline.

n

▪

n

Experience: 3–5 years of hands-on engineering experience in a collaborative, team-based environment with distributed messaging systems.

n

▪

n

Programming: Professional proficiency in Python, Java, or a similar programming/scripting language.

n

▪

n

Systems: Strong Unix/Linux fundamentals with the ability to troubleshoot application, deployment, environment, and runtime issues.

n

▪

n

Methodology: Familiarity with SDLC practices, CI/CD delivery models, change management, and Kubernetes-based deployments.

n

Technical Competencies

n

Candidates are not expected to be experts in every tool, but should bring strong hands-on experience across several of the following areas:

n

▪

n

AWS Services: Hands-on experience with AWS compute and migration patterns, including ECS, EKS, Lambda, CloudWatch, and related cloud services.

n

▪

n

Apache Kafka: Deep hands-on experience with Kafka architecture including brokers, topics, partitions, consumer groups, replication, and cluster operations.

n

▪

n

AWS MSK: Practical experience with Amazon Managed Streaming for Apache Kafka including cluster provisioning, configuration, monitoring, and security (IAM, mTLS, SASL/SCRAM).

n

▪

n

Migration Tooling: Experience with MirrorMaker 2, Confluent Replicator, or similar cross-cluster replication tools for data migration and offset synchronization.

n

▪

n

Cloud Migration: Practical understanding of on-prem to cloud migration strategies, workload assessment, testing, cutover, and post-migration validation.

n

▪

n





Kafka Ecosystem: Familiarity with Schema Registry, Kafka Connect, Kafka Streams, and related ecosystem components.

n

▪

n

Networking and Security: Understanding of VPC peering, PrivateLink, TLS encryption, IAM policies, and network connectivity patterns for hybrid Kafka architectures.

n

▪

n

Programming and Automation: Experience with Python, Java, Shell, or similar languages to build scripts, utilities, and automation frameworks for topic creation, ACL migration, and consumer group management.

n

▪

n

APIs and Integration: Working knowledge of RESTful APIs, API design, and API Gateway patterns.

n

▪

n

CI/CD and Version Control: Robust experience with GitLab, Jenkins, Maven, or similar tools supporting automated build, test, and deployment pipelines.

n

▪

n

Observability: Familiarity with monitoring, logging, alerting, and troubleshooting using tools such as CloudWatch, Prometheus, Grafana, Splunk, or equivalent platforms for Kafka/MSK cluster health.

n

▪

n

Automation Excellence: Ability to design repeatable automation and apply AI-assisted engineering approaches to improve migration scale, quality, and efficiency.

n

Core Competencies

n

▪

n

Exceptional analytical, troubleshooting, and debugging skills.

n

▪

n

Strong ownership mindset with the ability to drive work to closure and meet commitments.

n

▪

n

Clear written and verbal communication, including concise status updates, structured briefings, and proactive stakeholder management.

n

▪

n

Effective collaboration across application, infrastructure, platform, and global engineering teams.

n

▪

n

Ability to work constructively across time zones, build alignment, and resolve issues with urgency and professionalism.

n

▪

n

Strong integrity, sound judgment, and commitment to positive conduct and ethical decision-making.

n

▪

n

High energy,

📌 Kafka to AWS MSK Migration Engineer (Bengaluru)
🏢 Lorven Technologies
📍 Bengaluru