We're looking for Automation Test Engineers

Responsibilities:

• Design and execute comprehensive test strategies specifically for evaluating LLM responses. Focus heavily on measuring accuracy, ensuring long-term context retention, and drastically reducing or mitigating hallucinations.
• Rigorously test the underlying logic, complex state management, and tool-calling capabilities of autonomous AI agents across various edge cases.
• Develop, maintain, and scale custom automated testing frameworks from scratch using Python.
• Continuously evaluate AI models by seamlessly integrating custom testing and evaluation frameworks into existing CI/CD pipelines.
• Define and implement novel evaluation metrics for non-deterministic AI features where standard deterministic software testing (exact-match) falls short.
• Work closely with AI developers, data scientists, and product managers to establish quality benchmarks for AI-driven features.

Qualifications:

• Strong programming proficiency in Python, with an emphasis on writing clean, scalable, and maintainable automation scripts.




• Deep understanding of Large Language Model (LLM) architectures, prompt engineering, and model behaviors.
• Familiarity building or testing RAG (Retrieval-Augmented Generation) pipelines and agentic frameworks (e.g., LangChain, LlamaIndex).
• Proficiency in validating non-deterministic outputs where traditional exact-match assertions do not apply (e.g., semantic similarity, LLM-as-a-judge, or custom evaluation heuristics).
• Solid understanding of CI/CD tools (e.g., GitHub Actions, GitLab CI, Jenkins) and version control (Git).
• Robust analytical mindset with the ability to foresee edge cases in generative AI applications.
• Proficiency in AI evals, guardrails, and testing/automating AI agents, which are critical skills for the MAP team.

(Note: Relevant applicants will receive an email requesting additional information as part of the next stage. The application process will be completed directly

📌 Automation Test Engineer (Bengaluru)
🏢 Scoutit
📍 Bengaluru