About Hupo

We are an AI-native start-up building sales enablement products in the banking, financial services, and insurance industry; already trusted by dozens of enterprise customers (including Fortune 500 companies).

Role at a Glance

Title: QA Automation Engineer - AI
Team: Engineering
Employment Type: Full-time
Location: Remote (APAC)

Why This Role Matters

At Hupo, AI quality is the product - a wrong score, hallucinated feedback, or a broken voice turn is a customer-facing defect, and in a regulated BFSI context, a compliance risk. This role owns and scales the testing of Hupo’s AI pipelines, working closely with QA, Engineering, and Product as the platform grows.

It’s a hands-on, high-ownership role in a fast-moving environment - you’ll be a key part of deciding, with evidence, whether each release is safe to ship.

What You'll Do

- Build and maintain automated testing for AI voice and chat agents, from single conversational turns to full roleplay flows.

- Test LLM output quality - correctness, consistency, structured output, language fidelity, prompt regression - using evaluation harnesses (LLM-as-judge, golden datasets, tolerance-based assertions for non-determinism).

- Validate RAG pipelines: retrieval relevance, grounding/faithfulness, and answer quality.

- Test voice pipelines: STT/TTS accuracy and real-time, low-latency behavior across languages.

- Automate across UI/E2E, API, and AI output layers, and build CI/CD from scratch - lint, type-check, tests, evals, coverage gates, deployment checks.

- Own release quality: regression strategy, catching breakages early, and clear go/no-go calls.

- Cover database and load/performance testing; extend automation to desktop and mobile clients.

- Take over and extend existing QA automation, and strengthen shared frameworks, tooling, and QA processes/standards.

What Success Looks Like

Within your first few months, you'll:

- Ramp on Hupo’s existing QA automation suites and take clear ownership of them.

- Extend test coverage for LLM and voice-agent behavior,



including new evaluation harnesses for non-deterministic output.

- Contribute CI/CD improvements - coverage gates, deployment checks - that catch regressions before they reach customers.

You’ll also be expected to continuously:

- Adapt and expand impact as priorities shift

- Take ownership of new problems as they emerge

What We're Looking For

Must-Haves

- 5+ years as a QA Automation Engineer, with a proven track record testing AI systems - not just traditional software.

- Builder mindset: established testing frameworks, standards, and CI/CD-integrated automation from scratch.

- Hands-on experience testing conversational AI - voice and/or chat agents - on real, shipped projects.

- Strong grasp of LLMs, prompt engineering, and RAG - designing tests for nondeterministic output and evaluating retrieval/generation quality.

- Understanding of voice pipelines (speech-to-text, text-to-speech) and how to test them in automation.

- Hands-on with LLM evaluation/observability tooling - e.g. Langfuse, LangSmith, DeepEval, RAGAS.

- Proficient in UI/E2E (e.g. Playwright) and API test automation; experience building CI/CD pipelines from scratch.

- Comfort with database testing, load/performance testing, and desktop/mobile test automation.

- Solid Git discipline, basic cloud knowledge, and familiarity with the agile sprint lifecycle.

- Track record in fast-paced, quick-shipping environments - ramps quickly on unfamiliar systems, pragmatic about process.

Nice-to-Haves

- TypeScript - for reading the codebase and contributing test/automation code directly.

- Voice & real-time: LiveKit / WebRTC; STT/TTS via Azure Speech, Google Cloud Speech, Deepgram, or self-hosted Whisper.

- AI/ML infra & eval tooling: GPU inference (vLLM / TGI / Triton), Azure OpenAI, golden datasets, HITL pipelines, Braintrust, Promptfoo.

- QA & cloud tooling: Cypress, Postman, multi-cloud (AWS/Azure/GCP), Infrastructure-asCode (Terraform/Pulumi/CloudFormation), Docker/Kubernetes.

- Also useful: product analytics familiarity (e.g. PostHog), multilingual/localized testing experience, and production monitoring/error-tracking tools.

📌 Quality Assurance Engineer (India)
🏢 Hupo
📍 India