- We're looking for an AI QA Engineer to own quality across our growing portfolio of AI-powered products custom dashboards, AI CRMs, RFP platforms, lead-gen systems, agentic workflows, and automations delivered to agency clients across the US, UK, and AU.
- This role spans both the AI layer (prompts, agents, RAG, model outputs) and the full application stack around it (web app UI, APIs, databases, integrations, multi-tenant access).
- You'll catch issues that traditional QA misses and the ones that pure ML evaluation misses the messy middle where a working model still ships a broken product.

Key Responsibilities

- AI & LLM Quality Design test strategies for LLM-powered features, agents, RAG systems, and workflow automations (n8n, custom pipelines, MCP-based systems)
- Build and maintain golden datasets, evaluation suites, and prompt regression tests so model or prompt changes don't silently break production
- Validate output quality across factual accuracy, format adherence, tone, hallucination rate, instruction-following, and refusal behavior Test agentic and multi-step workflows end-to-end tool calls, handoffs, error recovery, loop detection, cost and latency under load Stress-test RAG pipelines: retrieval accuracy, chunking quality, context relevance, citation correctness
- Run adversarial testing prompt injection, jailbreak attempts, malformed inputs, edge-case client data Custom App & Platform Quality Own end-to-end QA for custom-built applications: dashboards, AI CRMs, client portals, and internal tools
- Functional, regression, cross-browser, and responsive testing across web UIs (Next.js, React, similar stacks) API testing for REST endpoints, webhooks, and third-party integrations (CRMs, email, calendars, payment,



file storage)
- Database and data-integrity testing especially for multi-tenant architectures where one client's data must never leak to another Auth and access-control testing: SSO, OAuth, role-based permissions, session handling
- Performance, load, and stability testing on production-bound features File upload, processing, and storage flows a recurring failure point in AI products Cross-Cutting
- Monitor production systems for drift, degradation, errors, and cost/token anomalies; set up observability and alerting
- Collaborate with AI engineers, full-stack developers, prompt designers, and delivery leads to catch issues before they hit clients
- Automate testing in Python and/or JavaScript; integrate into CI/CD pipelines Document findings, build QA dashboards, and contribute to E2M's internal AI QA playbook

Required Skills & Qualifications

- Bachelor's degree in Computer Science, IT, Engineering, or related field
- 3+ years QA experience covering both web applications and AI-enabled products, with at least 1 year hands-on testing LLM-powered features, AI workflows, or agent systems
- Robust Python skills for evaluation harnesses and automation; working JavaScript/TypeScript familiarity for web app testing
- Hands-on experience with at least one LLM evaluation framework: Promptfoo, DeepEval, RAGAs, LangSmith,



Braintrust, or similar Web app testing experience with Playwright, Cypress, or Selenium API testing with Postman, REST Assured, or equivalent
- Solid understanding of prompt engineering, prompt regression testing, and golden-dataset design Database fundamentals (SQL, basic schema reasoning) enough to verify data flows and spot multi-tenancy bugs
- Familiarity with LLM observability tools (Langfuse, Helicone, LangSmith) and token/cost/latency monitoring Git, CI/CD principles, and at least one cloud platform (AWS, GCP, or Azure)
- Working knowledge of vector databases, embeddings, and how RAG systems fail

Nice to Have

- Experience testing multi-tenant SaaS platforms or client-facing custom apps Workflow automation platforms (n8n, Make, Zapier) or agent frameworks
- Exposure to MCP (Model Context Protocol) servers or tool-calling architectures
- Familiarity with Supabase, Postgres, or similar backends commonly used in our stack
- Experience with Claude, OpenAI, or open-source LLM APIs in production Agency or client-services background you understand what "client-ready" means
- Prompt injection / red-teaming experience SOC 2 or compliance-testing exposure .

Soft Skills

- Sharp critical eye you assume AI output (and the app around it) is broken until you've verified it
- Comfortable working in ambiguity; AI QA doesn't have settled playbooks yet
- Clear written communication bug reports for AI failures need precise reproduction steps
- Collaborative across AI engineers, full-stack devs, delivery managers, and account leads
- Proactive about catching issues before they reach the client demo.

📌 AI QA Engineer (Ahmedabad)
🏢 Huptech HR Solutions
📍 Ahmedabad