Title QA Engineer

Location India (Remote)

Duration Contract 6 to 12 months

Job Description

Overview

The QA Engineer II plays a crucial role in ensuring the quality and reliability of our Conversational AI solutions. Unlike deterministic software, the AI agents under test are non-deterministic, so benchmarking emphasizes semantic/behavioral evaluation in addition to exact-match assertions.

The ideal candidate has a strong foundation in software testing methodologies, experience with test case design and execution, and familiarity with automation frameworks. This role is ideal for detail-oriented professionals who are eager to learn, contribute to test automation, and help drive continuous improvements in software quality.

Responsibilities

Agentic Benchmark Design: Generate, import, create, extend, and manage benchmarks for non-deterministic LLM agents, validating behavior and responses.
Functional & Regression Testing: Grow benchmarks and create current ones as new functionality or issues are uncovered.
Benchmarking Automation Support:



Contribute to streamlining the benchmarking process in alignment with product development work.
Collaboration: Work closely with developers, product managers, and senior QA engineers to ensure test coverage aligns with product requirements.
Defect Identification & Reporting: Log detailed defect reports and work with engineering teams to ensure timely resolution.
Process & Traceability: Maintain requirement-to-test traceability and document detailed test plans and coverage to support audits/reviews for OEM programs.
Release Reporting: Produce per-release quality reports as benchmarks evolve and grow over time.
Performance & Load Testing Support: Assist in load and performance testing using tools such as JMeter.
Agile Testing & Continuous Improvement: Participate in Agile development cycles, contributing to continuous process improvements.

Qualifications

Basic Qualifications (Non-Negotiable)

3+ years of experience in software/prod

📌 QA Engineer (India)
🏢 ArrowCore Group
📍 India