- Understand the design specifications.
- Follow the best practices for core product development.
- ​Data ingestion for distributed system with parallel processing, in Java, CPP or Golang for
- from cloud storage – Amazon S3/Azure Cloud Storage/Google Cloud Storage, relational database – Snowflake, Google BigQuery, PostgreSQL, Files, Kafka, data lakehouse (Iceberg)
- High-availability (HA) loading, cross-region replication
- Loading monitoring/error reporting
- Spark Connector
- Third-party management – Kafka, Kafka Connect

Requirements

- building distributed systems applying applying parallel processing.
- Any or of the following – Kafka, Zookeeper, Spark, Stream processing.
- Experience in Kafka Connect, Kafka stream, Kafka security and customization
- Spark connectors
- Event driven architectures.
- Agile development
- CICD

​

Nice to have:

· gRPC protocol experience and multi-threading

· Zookeeper/ ETCD/ Consul

· Familiarity with distributed consensus algorithms (Paxos / Raft)

· Docker, Kubernetes

​

📌 Principal/Senior Data Engineer : Product Development (India)
🏢 Agivant Technologies
📍 India