AI QA Engineer/Consultant

Remote

Experience:

5+ years in AI/ML, GenAI, Model QA/Validation, Software QA, Data Science, or Quality Engineering, preferably with experience in regulated or quality-critical environments.

Role Overview: Responsible for ensuring the quality, reliability, performance, and compliance of AI/ML, GenAI, LLM, and Agentic AI solutions within Pfizers pharmaceutical and manufacturing environment.

Key Responsibilities

- Develop and execute AI/ML model QA and evaluation strategies across the model lifecycle.
- Evaluate ML models using Accuracy, Precision, Recall, F1, ROC/AUC, Confusion Matrix, and other relevant metrics.
- Test models across datasets, edge cases, failure scenarios, and different operating conditions.
- Perform GenAI/LLM evaluation for accuracy, relevance, consistency, hallucination, groundedness, and response quality.
- Design and execute testing for AI agents and multi-agent workflows, including orchestration, tool calling, and output validation.
- Build test datasets, evaluation frameworks, test harnesses, and automated QA pipelines.
- Use Python, APIs, scripting, and automation for AI testing and evaluation.
- Work with LangChain, LangGraph, Langfuse and similar tools for agent development, tracing, observability, and evaluation.
- Identify bias, drift, hallucination,



instability, and model/application failures and perform root-cause analysis.
- Define test cases, user stories, acceptance criteria, quality thresholds, and production-readiness criteria.
- Collaborate with Data Scientists, ML Engineers, Software Engineers, Product Owners, and Business Stakeholders.
- Document test results, model limitations, risks, defects, and evaluation outcomes in an audit-ready format.
- Support Responsible AI, AI Governance, Data Integrity, and Quality-by-Design practices.

Must-Have Skills

- Machine Learning & Model Evaluation
- GenAI / LLM Evaluation
- Agentic AI & Multi-Agent Systems
- Precision, Recall, F1, ROC/AUC, Confusion Matrix
- Test Strategy & Test Case Design
- Python, APIs & Automation
- LangChain, LangGraph, Langfuse
- Prompt Engineering & LLM Testing
- AI Risk, Bias & Robustness Testing
- Model/AI Application Regression Testing
- Strong Analytical & Stakeholder Management Skills

Valuable to Have

- Pharmaceutical / Life Sciences experience
- GxP / CSV / Data Integrity knowledge
- Azure / AWS / GCP
- MLOps / LLMOps / ModelOps
- RAG & Vector Databases
- Responsible AI & AI Governance
- CI/CD and automated testing
- AI Red-Teaming / Adversarial Testing

📌 Artificial Intelligence Engineer (Bengaluru)
🏢 Siro Clinpharm
📍 Bengaluru