Assessment — Data Engineer (Mid-level), Consumer Research

Crawl a brand across many sources, then pipeline & enrich the data

Level: Data Engineer, ~4 yrs' experience (Consumer Research / AI Data). Effort: Core build ~1 focused day, plus a short design doc. Please don't sink a whole weekend — we weight design and evaluation as much as code. How you'll be judged: completion first, then depth — and, deliberately, the decisions you make on your own. We are not prescribing sources, tools, schema, or approach. What you choose, and why, is a large part of what we're assessing. Make your reasoning explicit in the design doc. One fairness note: you won't be penalised for avoiding paid APIs — you may mock/stub an external paid dependency as long as you clearly document it. Everything else, build for real.

The objective

Given a brand, build a pipeline that gathers public data about that brand from across the internet, lands it cleanly, and turns it into something an analyst or our AI could use to understand the brand's public perception.

Pick a real,



well-known consumer brand (or use one we assign). Build it as if it has to run daily and scale — you're the one who'll operate it.

What You Must Deliver

Crawl the brand across at least 5 distinct sources
Work out where meaningful public data about the brand lives, and crawl five or more genuinely different sources. Which sources you choose — and why — is a core part of the evaluation. (Five pages of the same site is not five sources.)
Handle each source's realities: pagination, rate limits, retries/backoff, and things breaking.
Make the crawl incremental (a re-run collects only what's current) and replayable (persist raw responses).
In the design doc: list the sources you picked, why each one, what it contributes, and what you'd add with more time.
Pipeline the data into a clean, queryable store
Unify data from all sources into one coherent model — you decide the schema and the store, and justify both.
Idempotent & incremental;

📌 Senior Data Engineer (Chennai)
🏢 Wonder World
📍 Chennai