# Agent Evaluation & Operations Engineer
**Location:** Pune, Maharashtra
**Work Mode:** Work From Office (WFO)(890)
**Experience:** 8–11 Years
**Salary Budget:** ₹8,00,000–₹18,00,000 per annum (₹8–18 LPA)
**Notice Period:** Immediate to 45 Days
**Preferred Candidates:** Local candidates based in Pune

## Role Overview

We are seeking an experienced Agent Evaluation & Operations Engineer to develop, evaluate, test, monitor, and support AI agents and LLM-powered applications across their development and production lifecycle.

The ideal candidate must have strong hands-on expertise in Python, LangChain, LangGraph, OpenAI, and Azure Databricks. The role involves agent orchestration, LLM evaluation, experiment lifecycle management, automated testing, observability, production monitoring, troubleshooting, and operational reliability.

This is a client-facing position involving regular interaction with clients and technical teams. Excellent verbal and written communication skills are mandatory, with no compromise on communication quality.

## Key Responsibilities

### 1. Agent Development & Orchestration
- Design, develop, integrate, and maintain AI agents using LangChain, LangGraph, and OpenAI-based technologies.
- Build Python-based services, REST APIs, and agent integration components using FastAPI and Pydantic.
- Work with agent orchestration frameworks such as OpenAI Agents SDK, Google ADK, and Pydantic-AI, where applicable.
- Support agent workflow development, integration, testing, and production operations.

### 2. LLM Evaluation & Experiment Lifecycle
- Design and implement evaluation frameworks to assess agent quality, response accuracy, reliability, consistency, and performance.
- Develop automated evaluation suites using pytest and custom pytest-based evaluations.
- Work with MLflow Evaluation, DeepEval, Ragas, and promptfoo for LLM and agent evaluation.
- Manage experiment execution, evaluation results, test iterations, and quality improvement cycles.
- Identify performance gaps and support continuous improvement of AI agent outputs.

### 3. Agent Observability & Tracing
- Implement tracing, monitoring, and observability for agent workflows and LLM applications.
- Use OpenTelemetry and OpenInference for telemetry and instrumentation.
- Work with Langfuse and LangSmith to trace agent execution, investigate failures, and analyze application behavior.
- Monitor agent performance and troubleshoot issues across application and integration layers.

### 4. Azure AI Stack
- Work with Azure OpenAI and Microsoft Foundry for AI application and agent-related workloads.
- Use Azure Monitor and Application Insights for telemetry, application monitoring,



diagnostics, and troubleshooting.
- Support operational monitoring and performance analysis of Azure-hosted AI solutions.

### 5. Azure Databricks & Data Operations
- Work with Azure Databricks for data processing, SQL-based analysis, and operational workflows.
- Use Unity Catalog for data governance and access management.
- Work with Databricks Workflows/Jobs and SQL Warehouses.
- Support data-related workflows required for AI agent evaluation and operations.

### 6. CI/CD, Testing & Version Control
- Maintain source code and version control using Git and GitLab.
- Integrate automated tests and evaluation suites into CI/CD pipelines.
- Support repeatable testing, deployment validation, and release quality checks.
- Troubleshoot build, test, and deployment issues.

### 7. Production Monitoring & Reliability
- Monitor applications and services using Grafana, Prometheus, Azure Monitor, Kibana, and alerting platforms.
- Investigate production incidents, analyze logs and telemetry, and perform root-cause analysis.
- Support incident response, issue resolution, performance monitoring, and operational improvements.
- Help improve system reliability, observability, and troubleshooting efficiency.

### 8. Cloud Infrastructure, Containers & Security
- Work with Docker, Kubernetes, and Azure Kubernetes Service (AKS) for containerized workloads.
- Understand infrastructure-as-code practices using Terraform and Bicep.
- Apply security and identity management practices using Microsoft Entra ID, managed identities, OAuth2, Azure Key Vault, and role-based access control (RBAC).
- Support secure application access, credential management, and appropriate authorization controls.

### 9. Client & Stakeholder Collaboration
- Collaborate regularly with clients, engineering teams, and other stakeholders.
- Communicate technical issues, evaluation outcomes, incident updates, and resolutions clearly.
- Participate in technical discussions and troubleshooting sessions.
- Maintain qualified communication and timely coordination across teams.

## Mandatory Skills / Must-Haves

The following skills are explicitly mandatory for this position:

1. Python
2. LangChain
3. LangGraph
4. OpenAI
5. Azure Databricks

Candidates must also demonstrate excellent communication, analytical thinking,



problem-solving, and troubleshooting capabilities appropriate for a client-facing engineering role.

## Nice-to-Have Skills

### Programming & APIs
- FastAPI
- REST APIs
- Pydantic
- pytest

### Agent Orchestration
- OpenAI Agents SDK
- Google ADK
- Pydantic-AI

### Agent Observability
- OpenTelemetry
- OpenInference
- Langfuse
- LangSmith

### LLM Evaluation & Experiment Lifecycle
- MLflow Evaluation
- DeepEval
- Ragas
- promptfoo
- Custom pytest evaluations

### Azure AI Stack
- Azure OpenAI
- Microsoft Foundry
- Azure Monitor
- Application Insights

### Databricks
- Unity Catalog
- Databricks Workflows/Jobs
- SQL Warehouses

### CI/CD & Version Control
- Git
- GitLab
- Automated testing pipelines

### Production Monitoring
- Grafana
- Prometheus
- Azure Monitor
- Kibana
- Alerting platforms

### Cloud & Containers
- Docker
- Kubernetes
- Azure Kubernetes Service (AKS)
- Terraform
- Bicep

### Security & Identity
- Microsoft Entra ID
- Managed identities
- OAuth2
- Key Vault
- Role-Based Access Control (RBAC)

## Candidate Screening Criteria

- **Experience:** 8–11 years.
- **Employment stability:** Minimum 2 years in an organization.
- **Notice period:** Immediate to 45 days.
- **Communication:** Excellent verbal and written communication is mandatory; no compromise.
- **Location:** Local Pune candidates are preferred.
- **Work arrangement:** Work From Office is mandatory.
- **Shift flexibility:** Candidates must be comfortable with the specified weekday, weekend, public holiday, and rotational standby schedule.
- **Compensation:** ₹8–18 LPA salary budget.

## Shift Schedule

All timings below are in Singapore Time (SGT).

**Weekday shifts**
- Shift 1: 11:30 AM–8:00 PM
- Shift 2: 6:00 PM–1:00 AM

**Weekend and Public Holiday shifts**
- Shift 1: 9:00 AM–5:00 PM
- Shift 2: 5:00 PM–1:00 AM

**Rotational Standby**
- 1:00 AM–9:00 AM

Candidates must be willing to participate in rotational shifts, weekend and public holiday coverage, and the 1:00 AM–9:00 AM standby rotation.

## Interview Process

1. Technical Interview – Round 1
2. Technical Interview – Round 2
3. Client Interview
4. HR Discussion

## Application Details

Interested candidates should share their updated resume along with:
- Current location
- Total experience
- Relevant experience in Python, LangChain, LangGraph, OpenAI, and Azure Databricks
- Current CTC
- Expected CTC
- Official notice period and earliest possible joining date
- Confirmation of willingness to work from office and in rotational shifts

📌 Agent Evaluation & Operations Engineer (Pune)
🏢 Umanist Staffing
📍 Pune