A fast-growing AI evaluation company is building expert-level benchmarks that test whether AI agents can complete real research work from start to finish. They need practicing linguists to supply what only hands-on research experience provides: authentic problems, real data, and a rigorous way to prove an answer is right.

You will turn analyses from your own linguistics work into challenging, terminal-based tasks. Each task includes defined input files, a working reference solution, and an automated check that verifies the result, such as a reconstructed proto-form that correctly derives the attested reflexes. If you work in historical linguistics, computational phonology, or another subfield where results can be objectively tested, this is a chance to find out whether AI agents can do what you do.

This is a remote, task-based contract. You are paid per approved task (see How to Apply for compensation details), and you choose how many tasks you take on.

What You’ll Do

- Design expert-level benchmark tasks based on real analyses you have performed, such as comparative reconstruction, sound-change modeling, phonological rule induction, or morphological analysis, that an AI agent must solve end to end.
- Specify each task precisely , including the files, data, and systems it runs on, so it is realistic, self-contained, and reproducible from the command line.
- Write a working reference solution in Python (or another language) showing the task can be solved end to end in roughly 60–75 minutes.
- Build automated checks that objectively verify correctness,



for example confirming that a reconstructed form and its sound laws reproduce the attested forms in the data.
- Document how errors are caught , including expected outputs, edge cases, and failure modes, so reviewers can validate each task.
- Refine tasks based on reviewer feedback , adjusting difficulty so each one is expert-level, hard for current AI agents, and grounded in real research practice.

What You Bring
- PhD (completed or advanced candidacy) or 3+ years of professional research in a linguistics subfield such as historical linguistics or computational phonology, with publications, a thesis, or tangible research output.
- 3+ years doing the analytical work yourself (not only studying, managing, or teaching it).
- Hands-on work within the last 3 years, and the ability to name the tools, datasets, and rules you used.
- You run analyses in code (Python, R) or directly in domain software.
- A clear sense of what a checkable result looks like in your subfield.
- Comfort with Python (or another language), the command line, and CSV/JSON files, APIs, or databases.

Nice to Have
- First-author papers with released code or data
- Teaching or peer-review experience in your subfield
- Working knowledge of Docker and Git
- Experience writing automated tests
- Experience with pandas for data analysis

Why This Role
- Your expertise sets the bar for how AI agents are evaluated on real linguistic research.
- Adaptable, remote, and self-paced. Take on as many tasks as fit your schedule.
- Paid per approved task , with clear acceptance criteria.
- Built on work you already know. Use the datasets, methods, and tools from your own research.

📌 Computational Linguistics Expert for AI Benchmarking (India)
🏢 Toptal
📍 India