Job Description
- Understand the design specifications.
- Follow the best practices for core product development.
- ​Data ingestion for distributed system with parallel processing, in Java, CPP or Golang for
- from cloud storage – Amazon S3/Azure Cloud Storage/Google Cloud Storage, relational database – Snowflake, Google BigQuery, PostgreSQL, Files, Kafka, data lakehouse (Iceberg)
- High-availability (HA) loading, cross-region replication
- Loading monitoring/error reporting
- Spark Connector
- Third-party management – Kafka, Kafka Connect

Requirements
- building distributed systems applying applying parallel processing.
- Any or of the following – Kafka, Zookeeper, Spark, Stream processing.
- Experience in Kafka Connect, Kafka stream, Kafka security and customization
- Spark connectors
- Event driven architectures.
- Agile development
- CICD

​
Nice to have:
· gRPC protocol experience and multi-threading
· Zookeeper/ ETCD/ Consul
· Familiarity with distributed consensus algorithms (Paxos / Raft)
· Docker, Kubernetes
​

📌 Principal/Senior Data Engineer (Hyderabad)
🏢 Agivant Technologies
📍 Hyderabad