Position:
Lead Data Platform Engineer (OpenSearch)
Job Description:
Role: Lead Data Platform Engineer (AWS | Kafka | OpenSearch)
Preferred Location: Ahmedabad, Pune, Indore
Years of Experience: 08 – 12 years
Notice Period – 0-30 Days (Preferred)

Introduction
Our team builds a data platform that reliably collects, processes, and retrieves large-scale IoT data based on AWS. We are seeking senior engineer who can design unobstructed flow even among billions of data records, push data storage performance to the limit, and technically lead service growth.

Key Responsibilities
High-availability data pipeline design: AWS MSK (Kafka), end-to-end data streaming architecture design and operation using Logstash
Storage optimization and sharding strategies: Optimizing large-scale search performance through OpenSearch's index design, sharding, and lifecycle (ISM) management
Data Modeling and Partitioning: Table partitioning and data modeling in PostgreSQL/MySQL, architecture guide for effective querying
Performance Tuning and Guidance:



Leading SQL optimization across services and spreading technical best practices to backend development teams
Release Management: Be the gatekeeper during key database / data pipeline release activities, create review checklist, review deployment details and validates details before deployment to ensure zero-error deployments without roll back needs.
Infrastructure automation: DB schema versioning (such as Flyway) and deployment process automation systems

Requirements
Over 7 years of experience in data engineering or related backend development
Extensive experience in operating infrastructure handling large-scale traffic in AWS cloud environments
Strong capabilities in building real-time data pipelines and troubleshooting using Kafka
Experience in OpenSearch (Elasticsearch) cluster design and sharding optimization is essential.
Deep understanding of the internal structure of RDBMS (PostgreSQL/MySQL) and ability to tune complex queries
Preferred

📌 Lead Data Platform Engineer (Vijayanagar)
🏢 Arrow Electronics
📍 Vijayanagar