Key Responsibilities

- Design and execute test strategies specifically for AI/ML models, LLM-based applications, and data pipelines.
- Develop automated test frameworks for model validation, regression testing, and performance

benchmarking.

- Evaluate model outputs for accuracy, consistency, relevance, hallucination, and bias across diverse inputs.
- Test RAG (Retrieval-Augmented Generation) pipelines, chatbots, recommendation systems, and other AI-

driven features.

- Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds.
- Build and maintain evaluation datasets, ground truth sets, and adversarial test cases.
- Monitor models in production for drift, degradation, and anomalous behavior.
- Validate data quality, data pipelines, and feature stores that feed AI systems.
- Document defects, edge cases, and failure patterns specific to AI behavior.
- Ensure AI systems meet ethical, fairness, and compliance standards (bias audits, explainability checks).

Required Experience

- 3–6 years of overall QA experience, with at least 1–2 years specialized in AI/ML quality assurance.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field.




- Robust proficiency in Python for test automation and data analysis.
- Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith).
- Hands-on experience with testing tools: Pytest, Selenium, Postman, or similar.
- Solid understanding of the ML lifecycle — training, validation, deployment, and production monitoring.
- Knowledge of data quality tools and pipeline testing (e.g., Great Expectations, dbt tests).
- Analytical and inquisitive mindset with red-team thinking to challenge model outputs effectively.
- Strong documentation, communication, and cross-functional collaboration skills.

Preferred Skills

- Experience with prompt engineering and red-teaming LLMs.
- Familiarity with MLOps platforms (e.g., MLflow, SageMaker, Vertex AI).
- Knowledge of vector databases and embedding quality evaluation.
- Understanding of AI safety, responsible AI principles, and fairness frameworks.
- Experience with A/B testing and shadow deployment strategies for AI models.

Pay: From ₹1,000,000.00 per year

Work Location: In person

📌 QA Engineer (India)
🏢 Straive
📍 India