# Agent Evaluation & Operations Engineer
•

Location:
*
* Pune, Maharashtra
• Work Mode:*
* Work From Office (WFO)(890)
•

Experience:
*
* 8–11 Years
• Salary Budget:*
* ₹8,00,000–₹18,00,000 per annum (₹8–18 LPA)
• Notice Period:*
* Immediate to 45 Days
• Preferred Candidates:*
* Local candidates based in Pune ## Role Overview We are seeking an experienced Agent Evaluation & Operations Engineer to develop, evaluate, test, monitor, and support AI agents and LLM-powered applications across their development and production lifecycle. The ideal candidate must have strong hands-on expertise in Python, LangChain, LangGraph, OpenAI, and Azure Databricks. The role involves agent orchestration, LLM evaluation, experiment lifecycle management, automated testing, observability, production monitoring, troubleshooting, and operational reliability. This is a client-facing position involving regular interaction with clients and technical teams. Excellent verbal and written communication skills are mandatory, with no compromise on communication quality. ##

Key Responsibilities
### 1. Agent Development & Orchestration
• Design, develop, integrate, and maintain AI agents using LangChain, LangGraph, and OpenAI-based technologies.
• Build Python-based services, REST APIs, and agent integration components using FastAPI and Pydantic.
• Work with agent orchestration frameworks such as OpenAI Agents SDK, Google ADK, and Pydantic-AI, where applicable.
• Support agent workflow development, integration, testing, and production operations. ### 2. LLM Evaluation & Experiment Lifecycle
• Design and implement evaluation frameworks to assess agent quality, response accuracy, reliability, consistency, and performance.
• Develop automated evaluation suites using pytest and custom pytest-based evaluations.
• Work with MLflow Evaluation, DeepEval, Ragas, and promptfoo for LLM and agent evaluation.
• Manage experiment execution, evaluation results, test iterations, and quality improvement cycles.
• Identify performance gaps and support continuous improvement of AI agent outputs. ### 3. Agent Observability & Tracing
• Implement tracing, monitoring, and observability for agent workflows and LLM applications.
• Use OpenTelemetry and OpenInference for telemetry and instrumentation.
• Work with Langfuse and LangSmith to trace agent execution, investigate failures, and analyze application behavior.
• Monitor agent performance and troubleshoot issues across application and integration layers. ### 4. Azure AI Stack
• Work with Azure OpenAI and Microsoft Foundry for AI application and agent-related workloads.
• Use Azure Monitor and Application Insights for telemetry, application monitoring, diagnostics, and troubleshooting.




• Support operational monitoring and performance analysis of Azure-hosted AI solutions. ### 5. Azure Databricks & Data Operations
• Work with Azure Databricks for data processing, SQL-based analysis, and operational workflows.
• Use Unity Catalog for data governance and access management.
• Work with Databricks Workflows/Jobs and SQL Warehouses.
• Support data-related workflows required for AI agent evaluation and operations. ### 6. CI/CD, Testing & Version Control
• Maintain source code and version control using Git and GitLab.
• Integrate automated tests and evaluation suites into CI/CD pipelines.
• Support repeatable testing, deployment validation, and release quality checks.
• Troubleshoot build, test, and deployment issues. ### 7. Production Monitoring & Reliability
• Monitor applications and services using Grafana, Prometheus, Azure Monitor, Kibana, and alerting platforms.
• Investigate production incidents, analyze logs and telemetry, and perform root-cause analysis.
• Support incident response, issue resolution, performance monitoring, and operational improvements.
• Help improve system reliability, observability, and troubleshooting efficiency. ### 8. Cloud Infrastructure, Containers & Security
• Work with Docker, Kubernetes, and Azure Kubernetes Service (AKS) for containerized workloads.
• Understand infrastructure-as-code practices using Terraform and Bicep.
• Apply security and identity management practices using Microsoft Entra ID, managed identities, OAuth2, Azure Key Vault, and role-based access control (RBAC).
• Support secure application access, credential management, and appropriate authorization controls. ### 9. Client & Stakeholder Collaboration
• Collaborate regularly with clients, engineering teams, and other stakeholders.
• Communicate technical issues, evaluation outcomes, incident updates, and resolutions clearly.
• Participate in technical discussions and troubleshooting sessions.
• Maintain skilled communication and timely coordination across teams. ## Mandatory Skills / Must-Haves The Following Skills Are Explicitly Mandatory For This Position
• Python
• LangChain
• LangGraph
• OpenAI
• Azure Databricks Candidates must also demonstrate excellent communication, analytical thinking, problem-solving, and troubleshooting capabilities appropriate for a client-facing engineering role.



## Nice-to-Have Skills ### Programming & APIs
• FastAPI
• REST APIs
• Pydantic
• pytest ### Agent Orchestration
• OpenAI Agents SDK
• Google ADK
• Pydantic-AI ### Agent Observability
• OpenTelemetry
• OpenInference
• Langfuse
• LangSmith ### LLM Evaluation & Experiment Lifecycle
• MLflow Evaluation
• DeepEval
• Ragas
• promptfoo
• Custom pytest evaluations ### Azure AI Stack
• Azure OpenAI
• Microsoft Foundry
• Azure Monitor
• Application Insights ### Databricks
• Unity Catalog
• Databricks Workflows/Jobs
• SQL Warehouses ### CI/CD & Version Control
• Git
• GitLab
• Automated testing pipelines ### Production Monitoring
• Grafana
• Prometheus
• Azure Monitor
• Kibana
• Alerting platforms ### Cloud & Containers
• Docker
• Kubernetes
• Azure Kubernetes Service (AKS)
• Terraform
• Bicep ### Security & Identity
• Microsoft Entra ID
• Managed identities
• OAuth2
• Key Vault
• Role-Based Access Control (RBAC) ## Candidate Screening Criteria
•

Experience:
8–11 years.
• Employment stability: Minimum 2 years in an organization.
• Notice period: Immediate to 45 days.
• Communication: Excellent verbal and written communication is mandatory; no compromise.
•

Location:
Local Pune candidates are preferred.
• Work arrangement: Work From Office is mandatory.
• Shift flexibility: Candidates must be comfortable with the specified weekday, weekend, public holiday, and rotational standby schedule.
• Compensation: ₹8–18 LPA salary budget. ## Shift Schedule All timings below are in Singapore Time (SGT). Weekday shifts
• Shift 1: 11:30 AM–8:00 PM
• Shift 2: 6:00 PM–1:00 AM Weekend and Public Holiday shifts
• Shift 1: 9:00 AM–5:00 PM
• Shift 2: 5:00 PM–1:00 AM Rotational Standby
• 1:00 AM–9:00 AM Candidates must be willing to participate in rotational shifts, weekend and public holiday coverage, and the 1:00 AM–9:00 AM standby rotation. ## Interview Process
• Technical Interview – Round 1
• Technical Interview – Round 2
• Client Interview
• HR Discussion ## Application Details Interested Candidates Should Share Their Updated Resume Along With
• Current location
• Total experience
• Relevant experience in Python, LangChain, LangGraph, OpenAI, and Azure Databricks
• Current CTC
• Expected CTC
• Official notice period and earliest possible joining date
• Confirmation of willingness to work from office and in rotational shifts

Skills:
rest apis,python,azure databricks,azure,langgraph,langchain,docker,sql,kubernetes,communication,openai,ci/cd,git

📌 Agent Evaluation (Pune)
🏢 Umanist NA
📍 Pune