Job Description:

We are looking for an AI Quality Engineer to evaluate AI-generated outputs, define quality metrics, and develop practical approaches for testing non-deterministic, LLM-based systems. Candidates from software testing, automation, data science, or machine learning backgrounds are encouraged to apply.

The most important qualification for this role is the ability to research and develop innovative methods for evaluating LLM-based systems, where traditional pass/fail testing may not be sufficient. Experience with DeepEval or similar evaluation frameworks is a plus, but not required.

Roles & Responsibilities:

LLM Evaluation & Quality Assurance

- Design and execute testing strategies for Generative AI and LLM-powered applications
- Define quality standards, evaluation criteria, and success metrics for AI-generated outputs
- Research and develop innovative approaches for evaluating non-deterministic AI systems
- Create test datasets, benchmark scenarios, and repeatable evaluation methodologies
- Identify and analyze quality issues such as hallucinations, inconsistencies, inaccuracies, and reliability concerns
- Develop processes to measure and track AI quality over time

Automation & Analysis

- Build and maintain automated testing and evaluation solutions
- Analyze evaluation results and recommend quality improvements
- Support integration of AI quality validation into existing development and testing processes




- Develop reports, dashboards, and metrics that communicate AI quality and performance trends

Collaboration & Innovation

- Work with product owners, engineers, data scientists, and business stakeholders to define quality expectations
- Drive continuous improvement through experimentation and data-driven analysis
- Stay current with emerging AI evaluation techniques, tools, and best practices
- Help establish repeatable QA practices for AI-enabled solutions

Must-Have Skills

- Background as a Data Scientist, QA Automation Engineer, or similar technical role
- Hands-on experience with LLM evaluation
- Proven ability to research and build innovative evaluation methods for non-deterministic systems
- Working knowledge of Python (or another language used for testing and data analysis)
- Strong analytical, research, and problem-solving skills; comfortable in an emerging field where best practices are still evolving
- Good written and verbal communication skills

Valuable to Have

- Experience with DeepEval or similar LLM evaluation frameworks
- Exposure to Generative AI, Machine Learning, or NLP
- Experience evaluating chatbots, copilots, or intelligent assistants
- Familiarity with prompt engineering or AI platforms (OpenAI, Anthropic, Google Gemini)
- Experience in Financial Services or another regulated industry

📌 Automation Test Engineer / Data Scientist (India)
🏢 Qylis.Ai
📍 India