About shift
shift is a flexible contribution platform where eligible contributors can record short videos of everyday tasks to help train robotics and AI systems. It records how physical work actually happens and turns it into training-ready data. shift is trusted by 25000+ workers, across 15+ countries and paid out over 15M USD to contributors and annotators. shift has been featured in leading publications like Forbes, BBC, Business Insider, CBS, The Verge, Recent York Post, Entrepreneur, Morning Brew, Gizmodo.
Our mission is simple: make participation in the AI economy accessible, flexible, and fair.
Join the shift.
About the role
You are the gold-standard authority for shift's egocentric training data. You own the golden sets, SOPs, and calibration system that annotators, reviewers, and an AI-first pipeline are all measured against and you partner with engineering to stand up new annotation workflows in quality control, hand tracking, object identification, and action labelling. As shift scales from thousands to millions of contributors,
you build the quality system that lets output grow faster than headcount without letting the 99% accuracy bar slip.
What you'll do
Own the quality system
Define and maintain gold-standard benchmark ("golden") sets for new and existing tasks, with explicit acceptance criteria and frame-level tolerance thresholds
Author, version-control, and continuously refine annotation SOPs and rubrics based on disagreement patterns, audit findings, and the evolving client spec
Design and run Inter-Annotator Agreement measurement (Cohen's/Fleiss' kappa, tolerance-based agreement for temporal boundaries) and produce regular quality reports
Analyze disagreement trends to root cause and recommend data-driven process improvements
Build and maintain edge-case libraries and example banks; adjudicate escalated edge cases and turn rulings into guideline updates
Conduct precision-based QC audits and validate pilot batches before production scaling
Sca
📌 Data annotation lead (India)
🏢 Shift
📍 India