Job highlights

- Keep consumers healthy: retries,batch item failures,dedup,dead-letter queues,and Mongo and Redis connection limits
- Solid MongoDB experience (aggregations,indexes,change data) . Hands-on use of Claude Code or a similar AI coding tool,with the discipline to verify AI output through tests and real data before shipping . Nice to have

Job Summary We're hiring a Data Engineer to own our data platform end to end: the pipelines that move data from MongoDB into ClickHouse, the accuracy of every number our customers see, and the performance and cost of ClickHouse itself.

You'll work on our live production stack and use AI-assisted development (Claude Code) as part of your everyday workflow. You'll be the first responder when something in your area breaks, so you'll care about root causes, not restarts.

Responsibilities

1. Data pipelines

- Run and improve the event flow from MongoDB through Kinesis, SQS and Lambda consumers, plus ClickPipes, into ClickHouse.
- Keep consumers healthy: retries, batch item failures, dedup, dead-letter queues, and Mongo and Redis connection limits.
- Fix failures quickly and find the root cause. A restart doesn't count as a fix.
- Keep dev, staging and prod schemas in sync, and handle DDL and schema initialisation safely. 2. Data accuracy and reconciliation

- Own reconciliation between MongoDB (source) and ClickHouse (reports) for orders, feedbacks, loyalty points, credits and comm reports.
- Investigate customer-reported mismatches (points balances, sales totals, undelivered counts, segment counts) and close each one with a root cause.





- Build checks that catch drift before customers do: daily row counts, sums and key metrics compared per brand.
- Get timezone, order_type and status handling right everywhere a metric is computed.

3. Moving to incremental processing

- Replace full recompute on every event with incremental updates, using ClickHouse materialized views and AggregatingMergeTree instead of repeated MongoDB aggregations.
- Ship migrations in small, switchable steps, each verified against the old path before cut-over.
- Retire legacy pipeline paths (Kinesis Lambda ClickHouse) once the replacements are proven correct. 4. ClickHouse optimisation

- Own schema design: ORDER BY, partition keys, granularity and engine choice, following ClickHouse best practices.
- Tune the slowest and most expensive queries weekly using system.query_log, and keep the ClickHouse Cloud bill within target.

- Prevent merge pressure: no forced whole-partition merges, sensible insert batching, healthy part counts.
- Make sure dashboard and export endpoints meet their latency targets.

On-call: Each of these four areas carries its own on-call responsibility. When it breaks, you're the first responder.

Must have

- Solid ClickHouse and analytical SQL (CTEs,



window functions, argMax, joins)

- Production experience running SQS or Kinesis pipelines in Node.js
- Solid MongoDB experience (aggregations, indexes, change data)
- Hands-on use of Claude Code or a similar AI coding tool, with the discipline to verify AI output through tests and real data before shipping

Nice to have

- CDC tools such as PeerDB or Debezium

- EMR Serverless or Spark

- Python for data checks
- Experience moving batch recompute to incremental processing

Our stack

- AI-assisted development: Claude Code
- Pipelines: AWS SQS, Kinesis Data Streams, Lambda, Step Functions
- Analytics database: ClickHouse Cloud, ClickPipes (CDC)

- Source database: MongoDB (Mongoose)
- Compute and deploy: AWS ECS Fargate, Serverless Framework, EventBridge Scheduler, S3, SSM
- Caching and queues: Redis, Bull

- Language: Node.js (JavaScript)
- Observability: Datadog, AWS CloudWatch

What success looks like

- Customer-reported data mismatches are rare, and when they happen they're closed with a root cause.

- Drift is caught by your checks before customers notice.
- Metrics run on incremental paths, and the old pipelines are retired.
- Dashboards and exports hit their latency targets, and the ClickHouse bill stays on target.

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Data Engineer (Pune)
🏢 Reelo
📍 Pune