Project Role : Large Language Model Architect

Project Role Description : Architect large language models (LLM) that can process and generate natural language. Design neural network parameters, trained on large quantities of unlabeled text data.

Must have skills : Large Language Models (LLMs), Virtual Agents, Generative AI

Good to have skills : NA
Minimum 7.5 year(s) of experience is required

Educational Qualification : 15 years full time education

Summary:
Pod: Pod 2 Agent Runtime
Reports to: Senior LLM Systems Engineer (Pod 2 Tech Lead)
Experience: 610 years
Location: Bangalore / US (flexible)
Note: Explore Internal

ROLE OVERVIEW: The Evaluation Engineer owns the platform''s quality gate the pre-deployment evaluation harness that every agent must pass before it reaches production, and the regression suite that catches regressions in platform behavior.

Roles Responsibilities:




Design and build the pre-deployment evaluation harness test suite, scoring framework, and pass/fail criteria
Define evaluation dimensions for agent quality: task completion accuracy, context retrieval relevance, guardrail compliance, tool call correctness
Build and maintain the platform regression suite
Partner with the Senior LLM Systems Engineer on valid agent invocation criteria
Establish the evaluation data strategy
Build tooling that makes evaluation results interpretable to domain teams

IDEAL PROFILE:
Has built evaluation frameworks or test infrastructure for ML or LLM systems in production has robust software engineering fundamentals understands LLM behavior well enough to design meaningful evaluation criteria.

Qualification 15 years full time education

📌 Large Language Model Architect (Bengaluru)
🏢 Accenture
📍 Bengaluru