# Agent Evaluation &

- Operations Engineer

- Location: Pune, Maharashtra
- Work Mode: Work From Office (WFO)(890)
- Experience: 8–11 Years
- Salary Budget: ₹8,00,000–₹18,00,000 per annum (₹8–18 LPA)
- Notice Period: Immediate to 45 Days
- Preferred Candidates: Local candidates based in Pune

## Role Overview We are seeking an experienced Agent Evaluation &

- Operations Engineer to develop, evaluate, test, monitor, and support AI agents and LLM-powered applications across their development and production lifecycle.

The ideal candidate must have strong hands-on expertise in Python, LangChain, LangGraph, OpenAI, and Azure Databricks. The role involves agent orchestration, LLM evaluation, experiment lifecycle management, automated testing, observability, production monitoring, troubleshooting, and operational reliability.

This is a client-facing position involving regular interaction with clients and technical teams. Excellent verbal and written communication skills are mandatory, with no compromise on communication quality.

## Key Responsibilities

### 1.

Agent

Development &

- Orchestration

- Design, develop, integrate, and maintain AI agents using LangChain, LangGraph, and OpenAI-based technologies.
- Build Python-based services, REST APIs, and agent integration components using FastAPI and Pydantic.
- Work with agent orchestration frameworks such as OpenAI Agents SDK, Google ADK, and Pydantic-AI, where applicable.
- Support agent workflow development, integration, testing, and production operations.

### 2. LLM Evaluation &
- Experiment Lifecycle

- Design and implement evaluation frameworks to assess agent quality, response accuracy, reliability, consistency, and performance.
- Develop automated evaluation suites using pytest and custom pytest-based evaluations.
- Work with MLflow Evaluation, DeepEval, Ragas, and promptfoo for LLM and agent evaluation.
- Manage experiment execution, evaluation results, test iterations, and quality improvement cycles.
- Identify performance gaps and support continuous improvement of AI agent outputs.

### 3.

Agent

Observability &
- Tracing

- Implement tracing, monitoring, and observability for agent workflows and LLM applications.
- Use OpenTelemetry and OpenInference for telemetry and instrumentation.
- Work with Langfuse and LangSmith to trace agent execution, investigate failures, and analyze application behavior.
- Monitor agent performance and troubleshoot issues across application and integration layers.

### 4. Azure AI Stack
- Work with Azure OpenAI and Microsoft Foundry for AI application and agent-related workloads.
- Use Azure Monitor and Application Insights for telemetry, application monitoring, diagnostics,



and troubleshooting.
- Support operational monitoring and performance analysis of Azure-hosted AI solutions.

### 5.

Azure

Databricks &
- Data Operations

- Work with Azure Databricks for data processing, SQL-based analysis, and operational workflows.
- Use Unity Catalog for data governance and access management.
- Work with Databricks Workflows/Jobs and SQL Warehouses.
- Support data-related workflows required for AI agent evaluation and operations.

### 6. CI/CD, Testing &
- Version Control

- Maintain source code and version control using Git and GitLab.
- Integrate automated tests and evaluation suites into CI/CD pipelines.
- Support repeatable testing, deployment validation, and release quality checks.
- Troubleshoot build, test, and deployment issues.

### 7.

Production

Monitoring &
- Reliability

- Monitor applications and services using Grafana, Prometheus, Azure Monitor, Kibana, and alerting platforms.
- Investigate production incidents, analyze logs and telemetry, and perform root-cause analysis.
- Support incident response, issue resolution, performance monitoring, and operational improvements.
- Help improve system reliability, observability, and troubleshooting efficiency.

### 8.

Cloud

Infrastructure, Containers &
- Security

- Work with Docker, Kubernetes, and Azure Kubernetes Service (AKS) for containerized workloads.
- Understand infrastructure-as-code practices using Terraform and Bicep.
- Apply security and identity management practices using Microsoft Entra ID, managed identities, OAuth2, Azure Key Vault, and role-based access control (RBAC).
- Support secure application access, credential management, and appropriate authorization controls.

### 9. Client &
- Stakeholder Collaboration

- Collaborate regularly with clients, engineering teams, and other stakeholders.
- Communicate technical issues, evaluation outcomes, incident updates, and resolutions clearly.
- Participate in technical discussions and troubleshooting sessions.
- Maintain qualified communication and timely coordination across teams.

## Mandatory Skills / Must-Haves The Following Skills Are Explicitly Mandatory For This Position

- Python
- LangChain
- LangGraph
- OpenAI
- Azure Databricks

Candidates must also demonstrate excellent communication, analytical thinking,



problem-solving, and troubleshooting capabilities appropriate for a client-facing engineering role. ## Nice-to-Have Skills

### Programming &

- APIs

- FastAPI
- REST APIs
- Pydantic
- pytest

### Agent Orchestration
- OpenAI Agents SDK
- Google ADK
- Pydantic-AI

### Agent Observability
- OpenTelemetry
- OpenInference
- Langfuse
- LangSmith

### LLM Evaluation &
- Experiment Lifecycle

- MLflow Evaluation
- DeepEval
- Ragas
- promptfoo
- Custom pytest evaluations

### Azure AI Stack
- Azure OpenAI
- Microsoft Foundry
- Azure Monitor
- Application Insights

### Databricks
- Unity Catalog
- Databricks Workflows/Jobs
- SQL Warehouses

### CI/CD &
- Version Control

- Git
- GitLab
- Automated testing pipelines

### Production Monitoring
- Grafana
- Prometheus
- Azure Monitor
- Kibana
- Alerting platforms

### Cloud &
- Containers

- Docker
- Kubernetes
- Azure Kubernetes Service (AKS)
- Terraform
- Bicep

### Security &
- Identity

- Microsoft Entra ID
- Managed identities
- OAuth2
- Key Vault
- Role-Based Access Control (RBAC)

## Candidate Screening Criteria
- Experience: 8–11 years.
- Employment stability: Minimum 2 years in an organization.
- Notice period: Immediate to 45 days.
- Communication: Excellent verbal and written communication is mandatory; no compromise.
- Location: Local Pune candidates are preferred.
- Work arrangement: Work From Office is mandatory.
- Shift flexibility: Candidates must be comfortable with the specified weekday, weekend, public holiday, and rotational standby schedule.
- Compensation: ₹8–18 LPA salary budget.

## Shift Schedule All timings below are in Singapore Time (SGT).

Weekday shifts

- Shift 1: 11:30 AM–8:00 PM
- Shift 2: 6:00 PM–1:00 AM

Weekend and Public Holiday shifts

- Shift 1: 9:00 AM–5:00 PM
- Shift 2: 5:00 PM–1:00 AM

Rotational Standby

- 1:00 AM–9:00 AM

Candidates must be willing to participate in rotational shifts, weekend and public holiday coverage, and the 1:00 AM–9:00 AM standby rotation. ## Interview Process
- Technical Interview –
- Round 1
- Technical Interview –
- Round 2
- Client Interview
- HR Discussion

## Application Details Interested Candidates Should Share Their Updated Resume Along With

- Current location
- Total experience
- Relevant experience in Python, LangChain, LangGraph, OpenAI, and Azure Databricks
- Current CTC
- Expected CTC
- Official notice period and earliest possible joining date
- Confirmation of willingness to work from office and in rotational shifts

Skills: rest apis,python,azure databricks,azure,langgraph,langchain,docker,sql,kubernetes,communication,openai,ci/cd,git

📌 Agent Evaluation & Operations Engineer (Pune)
🏢 Umanist NA
📍 Pune