19 Nov
|
LabVantage Solutions
|
Kolkata
19 Nov
LabVantage Solutions
Kolkata
Job Title : Evaluation & Diagnostic AI Lead PSO
Objective:
LabVantage Solutions is an industry leading provider of laboratory software products. Our products enable scientists and analysts across the globe to develop novel solutions, work effectively, and meet regulatory compliance. The Evaluation & Diagnostic AI Lead will lead the evaluation and diagnostic part of the Agentic Development Life Cycle (ADLC), making sure every agent we build is secure, resistant to prompt injection, protects sensitive and personal data, and is reliable. The candidate turns AI quality from opinion into evidence: measurable, repeatable and auditable.
This is a leadership role: the candidate will set the direction of the capability, adapt as the field evolves and, over time, deploy the Evaluation & Diagnostic capability in the India Capability Center (ICC) so that our customers receive the best, validated outcomes.
Roles and Responsibilities :
- Design and build the evaluation and diagnostic capability on Arize AX: agent tracing and observability, evaluation datasets, large language model (LLM)-as-judge and code-based evaluators, experiments, monitors and dashboards. Cover Retrieval-Augmented Generation (RAG) and agent-trajectory evaluations, and set regression gates in continuous integration and delivery (CI/CD) so no agent is released without passing its evaluations.
- Train and mentor ICC engineers on evaluation, diagnostics and secure AI practices, building toward a self-sufficient ICC Evaluation & Diagnostic capability.
- Define the evaluation and diagnostic strategy across the ADLC: what is measured, acceptance thresholds, and test coverage for security, reliability and data protection, including prompt-injection and jailbreak testing,
personally identifiable information (PII) and sensitive-data leakage checks, and alignment with the Open Worldwide Application Security Project (OWASP) Top 10 for LLM Applications.
- Own and maintain the ADLC Work Instruction for the evaluation and diagnostic stages, keeping it current, auditable and followed.
- Track and assess new evaluation, security, reliability and diagnostic methods and tools, and bring proven improvements into the capability.
Job Requirements :
- Evaluation & Diagnostics Experience: 2+ years of proven experience creating evaluation test cases, running diagnostics, and guiding customers to auditable, secure and reliable AI.
- Engineering & Implementation: 7+ years in software engineering and custom project implementation, preferably in regulated environments.
- Evaluation & Observability Tools: Hands-on experience with LangSmith or similar evaluation and diagnostic tools such as Langfuse, Arize Phoenix, Ragas, DeepEval, promptfoo, TruLens, Braintrust or OpenAI Evals. Arize AX experience is preferred.
- AWS Bedrock: Experience with the Amazon Bedrock offering, including Bedrock Model Evaluation and Bedrock Guardrails, is preferred.
- AI Security & Data Protection: Experience testing agents for prompt injection, jailbreaks and data leakage, guided by the OWASP Top 10 for LLM Applications and red-teaming tools such as Garak or PyRIT; PII detection and redaction; least-privilege tool access.
- Software Engineering: Strong fundamentals in Python, APIs, Git, automated testing and CI/CD.
What Will Make You Stand Out:
- A golden-dataset evaluation suite with LLM-as-judge scoring, evaluation regression gates in CI/CD.
- Work in GxP-regulated laboratories are robust pluses.
📌 Evaluation & Diagnostic AI Lead PSO (Kolkata)
🏢 LabVantage Solutions
📍 Kolkata