Job Description

A QA Engineer for AI Initiatives is responsible for ensuring the quality, reliability, fairness, and performance of AI/ML-powered products and systems. Unlike traditional QA, this role requires deep understanding of non-deterministic model behavior, data quality, and AI-specific failure modes such as hallucinations, bias, and model drift.

Key Responsibilities
- Design and execute test strategies specifically for AI/ML models, LLM-based applications, and data pipelines
- Develop automated test frameworks for model validation, regression testing, and performance benchmarking
- Evaluate model outputs for accuracy, consistency, relevance, hallucination, and bias across diverse inputs
- Test RAG (Retrieval-Augmented Generation) pipelines, chatbots, recommendation systems, and other AI-driven features
- Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds
- Build and maintain evaluation datasets, ground truth sets, and adversarial test cases
- Monitor models in production for drift, degradation, and anomalous behavior




- Validate data quality, data pipelines, and feature stores that feed AI systems
- Document defects, edge cases, and failure patterns specific to AI behavior
- Ensure AI systems meet ethical, fairness, and compliance standards (bias audits, explainability checks)

Required Skills & Qualifications
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field
- 3–6 years of QA experience, with at least 1–2 years in AI/ML quality assurance
- Strong proficiency in Python for test automation and data analysis
- Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith)
- Hands-on experience with testing tools: Pytest, Selenium, Postman, or similar
- Understanding of ML lifecycle — training, validation, deployment, and monitoring
- Knowledge of data quality tools and pipeline testing (Outstanding Expectations, dbt tests)

Nice to Have
- Experience wit

📌 Developer, AI Engineering (Bengaluru)
🏢 EisnerAmper
📍 Bengaluru