AI QA Engineer/Consultant

Remote

Experience:

5+ years in AI/ML, GenAI, Model QA/Validation, Software QA, Data Science, or Quality Engineering, preferably with experience in regulated or quality-critical environments.

Role Overview: Responsible for ensuring the quality, reliability, performance, and compliance of AI/ML, GenAI, LLM, and Agentic AI solutions within Pfizers pharmaceutical and manufacturing workplace.

Key Responsibilities
Develop and execute AI/ML model QA and evaluation strategies across the model lifecycle.
Evaluate ML models using Accuracy, Precision, Recall, F1, ROC/AUC, Confusion Matrix, and other relevant metrics.
Test models across datasets, edge cases, failure scenarios, and different operating conditions.
Perform GenAI/LLM evaluation for accuracy, relevance, consistency, hallucination, groundedness, and response quality.
Design and execute testing for AI agents and multi-agent workflows, including orchestration, tool calling, and output validation.
Build test datasets, evaluation frameworks, test harnesses, and automated QA pipelines.
Use Python, APIs, scripting, and automation for AI testing and evaluation.
Work with LangChain, LangGraph, Langfuse and similar tools for agent development, tracing, observability, and evaluation.
Identify bias, drift, hallucination,



instability, and model/application failures and perform root-cause analysis.
Define test cases, user stories, acceptance criteria, quality thresholds, and production-readiness criteria.
Collaborate with Data Scientists, ML Engineers, Software Engineers, Product Owners, and Business Stakeholders.
Document test results, model limitations, risks, defects, and evaluation outcomes in an audit-ready format.
Support Responsible AI, AI Governance, Data Integrity, and Quality-by-Design practices.

Must-Have Skills
Machine Learning & Model Evaluation
GenAI / LLM Evaluation
Agentic AI & Multi-Agent Systems
Precision, Recall, F1, ROC/AUC, Confusion Matrix
Test Strategy & Test Case Design
Python, APIs & Automation
LangChain, LangGraph, Langfuse
Prompt Engineering & LLM Testing
AI Risk, Bias & Robustness Testing
Model/AI Application Regression Testing
Solid Analytical & Stakeholder Management Skills

Valuable to Have
Pharmaceutical / Life Sciences experience
GxP / CSV / Data Integrity knowledge
Azure / AWS / GCP
MLOps / LLMOps / ModelOps
RAG & Vector Databases
Responsible AI & AI Governance
CI/CD and automated testing
AI Red-Teaming / Adversarial Testing

📌 Artificial Intelligence Engineer Bengaluru (India)
🏢 Siro Clinpharm
📍 India