AI QA Engineer (Jaipur)

AI QA Engineer (Jaipur)

16 Nov
|
Konstant
|
Jaipur

16 Nov

Konstant

Jaipur

Job highlights

- Hands-on experience testing LLM applications,RAG pipelines,or AI agents in production or pre-production environments
- Skill and Ability: 3+ years of experience in QA engineering,test automation,ML QA,AI evaluation,or MLOps
- Strong proficiency in Python and test frameworks such as Pytest,with experience building automated test suites

We are looking for an AI QA Engineer to ensure the reliability, safety, and quality of intelligent applications powered by LLMs, RAG pipelines, and AI agents. You will design and execute functional, non-functional, and AI-specific tests covering accuracy, hallucinations, groundedness, bias, security, prompt injection, and workflow correctness. AI QA engineers own evaluation strategy, test automation, golden datasets, and go/no-go recommendations for AI releases.

Requirements and Duties

Design test strategies for AI applications including LLMs, RAG systems, conversational AI, and agentic workflows. Build automated evaluation pipelines for accuracy, groundedness, context relevance, reasoning, tool-use correctness, and latency. Create and maintain golden datasets, test scenarios, and evaluation metrics for regression testing.

Implement LLM-as-a-Judge, schema validation, tracing, and calibrated scoring for multi-turn and long-horizon tasks. Test for hallucinations, bias, harmful content, policy violations, PII leaks, and brand drift using filters and guardrails. Conduct red-team testing for prompt injection, jailbreaking, data exfiltration, and adversarial inputs.

Embed evaluation harnesses into CI/CD pipelines and monitor production telemetry for behavioral drift and failures. Analyze model and integration failures to identify root causes, controls, and mitigations.



Translate business risks into measurable quality criteria, thresholds, and release recommendations.

Collaborate with engineers, product managers, and security teams on testability, observability, and governance.

Skill and Ability

3+ years of experience in QA engineering, test automation, ML QA, AI evaluation, or MLOps. Strong proficiency in Python and test frameworks such as Pytest, with experience building automated test suites. Hands-on experience testing LLM applications, RAG pipelines, or AI agents in production or pre-production environments.

Understanding of AI failure modes: hallucinations, grounding errors, prompt sensitivity, context limits, and tool misuse. Familiarity with AI evaluation tools and platforms such as MLflow, Galileo, PyRIT, LangSmith, or equivalent. Knowledge of security testing concepts including prompt injection, jailbreaking, and data leakage risks.

Experience working with CI/CD, containers, cloud environments, and observability tools. Robust analytical skills to interpret evaluation results and communicate risks clearly.

Experience with multi-agent systems, autonomous workflows, or long-horizon task evaluation. Knowledge of responsible AI, fairness testing, and regulatory compliance for AI systems.

Background in software development, data engineering, or ML engineering. Contributions to AI testing frameworks, open-source evaluation tools, or internal QA standards.

Konsatnt Infosolutions

Pvt Ltd

Information Technology & Service

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 AI QA Engineer (Jaipur)
🏢 Konstant
📍 Jaipur

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: ai qa engineer (jaipur) / jaipur