About shift

shift pays everyday workers to record short videos of the tasks they already do (in kitchens, workshops, job sites, bakeries, and warehouses) to help train the next generation of AI and robotics. We’ve paid out over $15M to 25,000+ workers across 15+ countries.

The role

We need someone to own the quality bar for shift's egocentric training data - not to work to a spec, but to write it. You'll own the golden sets, SOPs, and calibration system that every annotator, reviewer, and auto-labeling model is measured against, and you'll work directly with engineering to stand up new annotation workflows in hand tracking, object identification, and action labelling. As we scale from thousands of contributors to millions, you're the reason the 99% accuracy bar holds. Your ground truth becomes the benchmark, and the model is measured against it.

What you'll do
- Own the quality bar end-to-end: golden/benchmark sets, acceptance criteria, frame-level tolerance thresholds, and the audit loops that hold them
- Author, version, and continuously sharpen annotation SOPs, rubrics,



and edge-case rules as the client spec evolves
- Design and run inter-annotator agreement measurement (kappa/alpha, tolerance-based agreement on temporal boundaries) and publish regular quality reports
- Analyse disagreement patterns to root cause - guideline ambiguity vs. annotator error - and fix the system, not just the file
- Stand up the training and certification pipeline that brings current annotators and reviewers to the bar fast: onboarding, structured calibration sessions, targeted retraining
- Design human-in-the-loop workflows where models pre-label and humans review, correct, and escalate - so output grows faster than headcount
- Partner with engineering and product on new workflows: hand tracking (keypoints, left/right attribution, contact states), object identification (boxes/masks, ontology, naming consistency), action labelling (controlled vocabulary, segment boundaries)
- Specify

📌 Data Annotation Lead (Bengaluru)
🏢 Shift
📍 Bengaluru