Agent Evaluation Engineer (Freelance | WFH) (India)

Agent Evaluation Engineer (Freelance | WFH) (India)

15 Nov
|
begig
|
India

15 Nov

begig

India

About BeGig

BeGig is a tech freelancing platform connecting skilled technology professionals with companies and founders seeking specialized expertise on a flexible, project-based basis.

Your Chance

We’re looking for experienced Agent Evaluation Engineers who can evaluate, test, and improve the performance and reliability of AI agents and agentic systems. You’ll work on designing evaluation frameworks, testing multi-step agent workflows, measuring task accuracy, and identifying failures to ensure AI agents perform consistently and safely in real-world applications.

What You’ll Do

- Design and implement evaluation frameworks for AI agents and multi-agent systems
- Develop test datasets, benchmarks, and evaluation scenarios for agentic workflows
- Evaluate agent performance across task completion, accuracy, reasoning, and reliability
- Test tool calling, function calling, planning, memory, and multi-step execution
- Build automated evaluation pipelines for LLM-powered agents
- Assess AI-generated responses for factual accuracy, relevance, consistency, and hallucinations
- Evaluate RAG pipelines, retrieval quality, and context relevance
- Identify failure patterns, edge cases, and weaknesses in agent behavior
- Implement evaluation metrics such as task success rate, precision, recall, latency, and cost
- Conduct regression testing to ensure improvements do not affect existing agent capabilities
- Evaluate agent safety, guardrails, instruction following, and resistance to prompt injection
- Build monitoring and observability systems to track agent performance in production
- Collaborate with AI engineers, ML engineers,



and product teams to improve agent quality

What We’re Looking For

- Strong experience in AI/LLM evaluation, agent testing, or AI quality engineering
- Proficiency in Python and experience with automated testing frameworks
- Hands-on experience evaluating LLM-based applications and AI agents
- Understanding of agentic architectures, multi-agent systems, and orchestration workflows
- Familiarity with frameworks such as LangChain, LangGraph, CrewAI, or AutoGen
- Experience with evaluation tools such as Ragas, DeepEval, LangSmith, TruLens, or similar
- Understanding of LLM evaluation metrics, benchmarking, and model performance assessment
- Experience testing tool calling, function calling, memory, planning, and agent execution
- Knowledge of RAG evaluation, embeddings, semantic search, and retrieval metrics
- Experience with prompt engineering, structured outputs, and automated test case generation
- Understanding of hallucination detection, bias assessment, and AI safety evaluation
- Familiarity with LLM observability, tracing, and production monitoring
- Experience with CI/CD and automated evaluation pipelines is a plus
- Strong analytical thinking, attention to detail, and problem-solving skills

Why BeGig

- Flexible freelance and project-based opportunities
- Work with startups and enterprises across different industries
- Get matched with relevant projects based on your skills and experience
- Access to a trusted network of technology professionals and clients

Apply now to join BeGig as an Agent Evaluation Engineer and help build reliable, accurate, and trustworthy AI agent systems. ?

📌 Agent Evaluation Engineer (Freelance | WFH) (India)
🏢 begig
📍 India

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: agent evaluation engineer (freelance | wfh) (india) / india

Subscribe to this job alert:

Get the latest job offers by email for: agent evaluation engineer (freelance | wfh) (india) / india