About BeGig
BeGig is a tech freelancing platform connecting skilled technology professionals with companies and founders seeking specialized expertise on a flexible, project-based basis.
Your Chance
We’re looking for experienced Agent Evaluation Engineers who can evaluate, test, and improve the performance and reliability of AI agents and agentic systems. You’ll work on designing evaluation frameworks, testing multi-step agent workflows, measuring task accuracy, and identifying failures to ensure AI agents perform consistently and safely in real-world applications.
What You’ll Do
- Design and implement evaluation frameworks for AI agents and multi-agent systems
- Develop test datasets, benchmarks, and evaluation scenarios for agentic workflows
- Evaluate agent performance across task completion, accuracy, reasoning, and reliability
- Test tool calling, function calling, planning, memory, and multi-step execution
- Build automated evaluation pipelines for LLM-powered agents
- Assess AI-generated responses for factual accuracy, relevance, consistency, and hallucinations
- Evaluate RAG pipelines, retrieval quality, and context relevance
- Identify failure patterns, edge cases, and weaknesses in agent behavior
- Implement evaluation metrics such as task success rate, precision, recall, latency, and cost
- Conduct regression testing to ensure improvements do not affect existing agent capabilities
- Evaluate agent safety, guardrails, instruction following, and resistance to prompt injection
- Build monitoring and observability systems to track agent performance in production
- Collaborate with AI engineers, ML engineers,
and product teams to improve agent quality
What We’re Looking For
- Strong experience in AI/LLM evaluation, agent testing, or AI quality engineering
- Proficiency in Python and experience with automated testing frameworks
- Hands-on experience evaluating LLM-based applications and AI agents
- Understanding of agentic architectures, multi-agent systems, and orchestration workflows
- Familiarity with frameworks such as LangChain, LangGraph, CrewAI, or AutoGen
- Experience with evaluation tools such as Ragas, DeepEval, LangSmith, TruLens, or similar
- Understanding of LLM evaluation metrics, benchmarking, and model performance assessment
- Experience testing tool calling, function calling, memory, planning, and agent execution
- Knowledge of RAG evaluation, embeddings, semantic search, and retrieval metrics
- Experience with prompt engineering, structured outputs, and automated test case generation
- Understanding of hallucination detection, bias assessment, and AI safety evaluation
- Familiarity with LLM observability, tracing, and production monitoring
- Experience with CI/CD and automated evaluation pipelines is a plus
- Strong analytical thinking, attention to detail, and problem-solving skills
Why BeGig
- Flexible freelance and project-based opportunities
- Work with startups and enterprises across different industries
- Get matched with relevant projects based on your skills and experience
- Access to a trusted network of technology professionals and clients
Apply now to join BeGig as an Agent Evaluation Engineer and help build reliable, accurate, and trustworthy AI agent systems. ?
📌 Agent Evaluation Engineer (Freelance | WFH) (India)
🏢 begig
📍 India