Remote | Full-time | ₹9L – ₹15L/year
About the Role
We're looking for a Senior Software QA Engineer to be the quality backbone of our agentic AI systems. This role is less about writing test scripts and more about deep session review — analyzing how our AI agents behave in the real world, spotting patterns in failures, and turning those insights into engineering action. You'll also play an ongoing role in strengthening our AI Eval framework and Smartflows, helping both evolve alongside the systems they're built to test.
What You'll Do
Review AI Sessions: Regularly review agent/LLM session transcripts and logs to evaluate output quality, reasoning correctness, and tool-use behavior.
Pattern-Level Analysis: Analyze session data across volume to identify recurring failure patterns — bad tool selection, context drift, hallucination clusters, agent loops, latency spikes, prompt regressions — rather than one-off bugs.
Create Engineering Tickets: Translate findings into clear, well-documented tickets with reproduction context, impact assessment, and priority,



so engineering can act on them efficiently.
QA Verification Loop: Once engineering ships a fix, verify it against the original issue and re-check for regressions before sign-off and deployment.
Improve AI Eval & Smartflows: Continuously refine our AI Eval framework and Smartflows based on session findings — updating rubrics, scoring criteria, and flow logic so they keep pace with real-world agent behavior.
Process Improvement: Identify systemic gaps in the QA and engineering workflow and propose concrete improvements — better logging/observability, review cadences, ticket templates, evaluation rubrics, or automation opportunities.
Build Review Frameworks: Help define what "positive" looks like for agent behavior — success criteria, scoring rubrics, or checklists that make session review consistent across the team.
Cross-functional Collaboration: Work closely with engineers, Customer Success, and Internal QA to surface rea

📌 Senior Software QA Engineer — Agentic AI Systems (India)
🏢 Readyly
📍 India