5–8 years in QA automation, with 1–3 years in GenAI / API-based testing.

Key Responsibilities

Develop and maintain automated evaluation pipelines.

Implement evaluation scripts using Python frameworks (e.g., DeepEval, custom frameworks)

Integrate LLM/Chatbot APIs and agent workflows into evaluation pipelines

Execute dataset-driven evaluations and capture and process responses.

Support manual test scenario execution and validation

Assist in dataset creation and enrichment

Generate evaluation reports, and logs

Debug and troubleshoot execution issues.

Enable CI/CD integration for continuous evaluation.

Key Skills

Core GenAI Evaluation Skills

Experience with evaluation frameworks (e.g., DeepEval or Arize)

Understanding of LLM-as-a-Judge (G-Eval) methodology

Robust prompt engineering and evaluation design skills

Experience in manual evaluation of LLM outputs

Test Strategy

JD for GenAI Evaluation QA

Technical Skills

Strong programming in Python

Experience in API testing and integration





Proficiency in JSON handling, parsing, and data processing

Automation framework development/integration.

Knowledge of logging, reporting, and debugging tools

Agent Manual Testing & Dataset Skills:

Experience in

o Test scenario creation for GenAI use cases o Manual validation of LLM responses (qualitative assessment)

o Dataset creation and curation o Writing expected outputs or golden answers.

Ability to design edge cases, negative scenarios and adversarial inputs (prompt injection, jailbreaks)

Domain & QA Skills:

Strong foundation in software testing principles:

o Functional, integration, regression testing

Experience in test design, defect tracking, and reporting.

Strong analytical and problem-solving skills.

Conversational AI testing experience.

Understanding of AI agent behavior, workflows, and edge cases.

📌 Gen AI Testing - Sr. Engineer (India)
🏢 Sparix Global
📍 India