Head of Research (Mumbai)

Head of Research (Mumbai)

22 Nov
|
Employ
|
Mumbai

22 Nov

Employ

Mumbai

Title
- Head of Research
- Benchmarks Experience
- 1
- 4 years Location
- Mumbai (5 days, onsite) Job Role: Lead research and production of coding-focused benchmarks built around real AI failure cases.

What you'll do
• Build multiple benchmarks, with a repeatable task-design, validation and review process that keeps quality high.
• Lead our current benchmarking projects with AI labs.
• Study agent runs, group the failures and turn them into objectively graded tasks.
• Define what makes an evaluation useful: correctness, objective grading and difficulty.
• Guard against answer leakage and gaming, and measure genuine improvement.
• Guide the researchers who write and review tasks. Non-negotiable You have personally worked on building one or more benchmarks. Sole ownership is not required, but your own contribution must be clear, with an artifact or public work we can inspect. Technical requirements Must-haves
• Strong Python for data processing and evaluation pipelines: pandas, JSON/JSONL at scale, scripting and automation.




• Hands-on experience running LLMs or coding agents against graded tasks through an evaluation harness.
• Working knowledge of statistics for evaluation: sampling, significance, contamination and train/test separation.
• Comfort with git-based workflows and code review.
• Working knowledge of Docker for reproducing agent runs. Nice-to-haves
• Familiarity with benchmark frameworks such as Terminal-Bench, SWE-bench, HELM or lm
- evaluation-harness.
• Exposure to RL or post-training evaluation loops.
• Prompt engineering and agent framework experience (tool use, sandboxes).
• Contributions to open-source evals or published benchmark work. Solid skills Coding-agent evals such as Terminal-Bench and SWE-bench style tasks; experimental judgment (what a benchmark does and does not measure); reproducible pipelines; quality control across many outputs. Interested candidates please apply using this link: https://itszia.ai/roles/head-of-research-benchmarks-o65pj08qwet0

📌 Head of Research (Mumbai)
🏢 Employ
📍 Mumbai

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: head of research (mumbai) / mumbai

Subscribe to this job alert:

Get the latest job offers by email for: head of research (mumbai) / mumbai