What you'll do:

- Design and run evaluations for an AI agent: test sets, LLM-as-judge and human review,

regression checks before each release

- Build and tune guardrails and moderation for a regulated domain (credit and financial

products), working with Compliance on policy

- Write, test and iterate on prompts, using trace data to find and fix failure patterns
- Develop MCP servers and tool integrations that give the agent access to internal data and

services

- Fine-tune and evaluate models on specialized domain data, including small language

models (SLMs) for targeted tasks

- Prototype with System One models (for example, Jev and Laya) for fast classification,

routing and verification steps

- Monitor production quality, latency and cost in Langfuse, and turn findings into concrete

fixes

Required skills and experience

- LLM and agent evaluation

Building eval datasets and scoring frameworks, including LLM-as-judge

Measuring agent behavior across multi-turn conversations and tool calls

- Guardrails and moderation

Input and output moderation,



prompt-injection and jailbreak defenses

Tuning precision and recall of safety classifiers against real traffic

- Prompt engineering

Structured outputs, tool-use prompting, and systematic prompt iteration driven by

evals

- Cloud and LLM platform

AWS, especially Amazon Bedrock (Converse API, model selection, service tiers)

Langfuse or a similar LLM observability tool: tracing, scores, datasets

- Engineering fundamentals

Robust Python, REST APIs (FastAPI or similar), unit testing, Git-based workflows

Strongly preferred

- MCP (Model Context Protocol) server development
- Fine-tuning LLMs on specialized domains (LoRA or full fine-tuning), including data

preparation and evaluation

- Small language models (SLMs): selection, distillation, deployment
- System One models such as Jev and Laya
- Experience in financial services or another regulated industry

📌 Applied AI Engineer (Coimbatore)
🏢 NexisSpark Tech Solutions
📍 Coimbatore