Key Responsibilities

▪

Assess on-prem Kafka clusters and perform discovery across topics, partitions, consumer groups, throughput patterns, retention policies, and inter-service dependencies to determine migration readiness.

▪

Analyze workloads based on message volume, business criticality, ordering guarantees, latency requirements, and consumer group complexity to determine appropriate MSK cluster placement and configuration.

▪

Plan and provision required AWS MSK infrastructure capacity including broker sizing, storage, networking, and security configurations to support migration, testing, and production execution.

▪

Execute UAT and production parallel testing using Mirror Maker 2 or similar replication tooling, compare message delivery outcomes, capture evidence, and troubleshoot discrepancies through closure.

▪

Perform performance tuning and optimization of MSK clusters to ensure migrated workloads are stable, scalable, efficient, and production ready.

▪

Partner with application engineering, infrastructure, and platform teams to finalize migration plans, consumer/producer cutover approach, validation criteria, and rollback considerations.

▪

Manage release readiness, execute production cutover activities including consumer group migration, offset synchronization, and complete post-release checkout and validation procedures.

▪

Support legacy on-premises Kafka cluster decommissioning after successful migration and validation.

▪

Design, enhance, and implement automation frameworks and AI-assisted solutions to enable repeatable, efficient, large-scale Kafka-to-MSK migrations.

▪

Document migration procedures, operational learnings, risks, and best practices to improve the factory execution model.

Required Qualifications

Basic Qualifications

▪

Education: Bachelor's or Master's degree in Computer Science, Engineering, Applied Mathematics,



or a related quantitative discipline.

▪

Experience: 3–5 years of hands-on engineering experience in a collaborative, team-based environment with distributed messaging systems.

▪

Programming: Professional proficiency in Python, Java, or a similar programming/scripting language.

▪

Systems: Strong Unix/Linux fundamentals with the ability to troubleshoot application, deployment, environment, and runtime issues.

▪

Methodology: Familiarity with SDLC practices, CI/CD delivery models, change management, and Kubernetes-based deployments.

Technical Competencies

Candidates are not expected to be experts in every tool, but should bring solid hands-on experience across several of the following areas:

▪

AWS Services: Hands-on experience with AWS compute and migration patterns, including ECS, EKS, Lambda, Cloud Watch, and related cloud services.

▪

Apache Kafka: Deep hands-on experience with Kafka architecture including brokers, topics, partitions, consumer groups, replication, and cluster operations.

▪

AWS MSK: Practical experience with Amazon Managed Streaming for Apache Kafka including cluster provisioning, configuration, monitoring, and security (IAM, mTLS, SASL/SCRAM).

▪

Migration Tooling: Experience with Mirror Maker 2, Confluent Replicator, or similar cross-cluster replication tools for data migration and offset synchronization.

▪

Cloud Migration: Practical understanding of on-prem to cloud migration strategies, workload assessment, testing, cutover, and post-migration validation.

▪





Kafka Ecosystem: Familiarity with Schema Registry, Kafka Connect, Kafka Streams, and related ecosystem components.

▪

Networking and Security: Understanding of VPC peering, Private Link, TLS encryption, IAM policies, and network connectivity patterns for hybrid Kafka architectures.

▪

Programming and Automation: Experience with Python, Java, Shell, or similar languages to build scripts, utilities, and automation frameworks for topic creation, ACL migration, and consumer group management.

▪

APIs and Integration: Working knowledge of RESTful APIs, API design, and API Gateway patterns.

▪

CI/CD and Version Control: Strong experience with Git Lab, Jenkins, Maven, or similar tools supporting automated build, test, and deployment pipelines.

▪

Observability: Familiarity with monitoring, logging, alerting, and troubleshooting using tools such as Cloud Watch, Prometheus, Grafana, Splunk, or equivalent platforms for Kafka/MSK cluster health.

▪

Automation Excellence: Ability to design repeatable automation and apply AI-assisted engineering approaches to improve migration scale, quality, and efficiency.

Core Competencies

▪

Exceptional analytical, troubleshooting, and debugging skills.

▪

Strong ownership mindset with the ability to drive work to closure and meet commitments.

▪

Clear written and verbal communication, including concise status updates, structured briefings, and proactive stakeholder management.

▪

Effective collaboration across application, infrastructure, platform, and global engineering teams.

▪

Ability to work constructively across time zones, build alignment, and resolve issues with urgency and professionalism.

▪

Strong integrity, sound judgment, and commitment to good conduct and ethical decision-making.

▪

High energy,

📌 Kafka to AWS MSK Migration Engineer (India)
🏢 Lorven Technologies
📍 India