Job Description

A QA Engineer for AI Initiatives is responsible for ensuring the quality, reliability, fairness, and performance of AI/ML-powered products and systems. Unlike traditional QA, this role requires deep understanding of non-deterministic model behavior, data quality, and AI-specific failure modes such as hallucinations, bias, and model drift.

Key Responsibilities

Design and execute test strategies specifically for AI/ML models, LLM-based applications, and data pipelines

Develop automated test frameworks for model validation, regression testing, and performance benchmarking

Evaluate model outputs for accuracy, consistency, relevance, hallucination, and bias across diverse inputs

Test RAG (Retrieval-Augmented Generation) pipelines, chatbots, recommendation systems, and other AI-driven features

Collaborate with data scientists and ML engineers to define acceptance criteria and quality thresholds

Build and maintain evaluation datasets, ground truth sets, and adversarial test cases

Monitor models in production for drift, degradation, and anomalous behavior

Validate data quality, data pipelines, and feature stores that feed AI systems

Document defects, edge cases, and failure patterns specific to AI behavior

Ensure AI systems meet ethical, fairness, and compliance standards (bias audits, explainability checks)

Required Skills & Qualifications





Bachelor's or Master's degree in Computer Science, Engineering, or a related field

3–6 years of QA experience, with at least 1–2 years in AI/ML quality assurance

Strong proficiency in Python for test automation and data analysis

Familiarity with LLM evaluation frameworks (e.g., RAGAS, DeepEval, Promptfoo, LangSmith)

Hands-on experience with testing tools: Pytest, Selenium, Postman, or similar

Understanding of ML lifecycle — training, validation, deployment, and monitoring

Knowledge of data quality tools and pipeline testing (Great Expectations, dbt tests)

Nice to Have

Experience with prompt engineering and red-teaming LLMs

Familiarity with MLOps platforms (MLflow, SageMaker, Vertex AI)

Knowledge of vector databases and embedding quality evaluation

Understanding of AI safety, responsible AI principles, and fairness frameworks

Experience with A/B testing and shadow deployment strategies

Soft Skills

Analytical and inquisitive mindset — comfortable challenging model outputs

Ability to think like both a user and an adversary (red-team thinking)

Robust documentation and communication skills

Collaborative approach with data science, engineering, and product teams

High attention to detail with a quality-first attitude

Preferred Location: Bangalore

📌 Senior Developer, Artificial Engineering (Bengaluru)
🏢 EisnerAmper
📍 Bengaluru