About The Job Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include Benchmark , General Catalyst , Peter Thiel , Adam D'Angelo , Larry Summers , and Jack Dorsey .
Position: Shopify Storefront Agent Evaluator
Type: Contract Compensation: $40–$50/hour Location: Remote Commitment: up to 5 hours/week Role Responsibilities
- Evaluate the Storefront Agent by simulating real shopper interactions, including product discovery, comparisons, and inquiries about sizing, stock, shipping, and returns.
- Grade responses based on Relevance, Accuracy, and Safety to ensure they meet shopper expectations and adhere to store policies.
- Provide concise feedback on any failures to ensure clarity for team members not present during evaluation.
- Work independently to deliver precise and repeatable judgments on agent performance.
Qualifications Must-Have
- Consumer e-commerce fluency — you shop online regularly and understand effective product recommendations.
- Careful, consistent judgment — apply the rubric uniformly across responses.
- Clear, concise written English — articulate reasons for response failures in a sentence or two.
Preferred
- Hands-on Shopify merchant experience — familiarity with shopper behavior on Shopify stores.
- Experience using AI shopping assistants or chat agents — understand common failure points.
- Prior work evaluating, red-teaming, or annotating AI model outputs.
- Familiarity with retail operations: catalog and variant structure, inventory, shipping, and returns policy.
Application Process (Takes 20–30 mins to complete)
- Upload resume
- AI interview based on your resume
- Submit form
Resources & Support
- For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
- For any help or support, reach out to:
[email protected]
PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this prospect. ,
📌 E-commerce Evaluation Specialist (India)
🏢 Mercor
📍 India