AI Ops Engineer

ETS AI | GenAI Observability & Operations

Role Purpose

- Own post-deployment observability, reliability, and operational health for ETS AI agents across Copilot Studio and pro-code Azure applications.

- Replace manual, Power BI-centered monitoring with automated dashboards, proactive alerts, and actionable production telemetry for enterprise AI workloads.

Key Responsibilities

- Instrument agents, workflows, and supporting services using OpenTelemetry and distributed tracing across user requests, model calls, retrieval steps, tool execution, and agent workflow stages.

- Build operational dashboards using Azure Monitor, Application Insights, Log Analytics, and KQL; integrate Copilot Studio and Microsoft Foundry telemetry where available.

- Define and operationalize alerts, thresholds, SLIs, and SLOs covering availability, latency, error rates, task completion, retrieval quality, token consumption, and inference cost.

- Correlate agent runs, distributed traces, evaluation results, logs, and production incidents to identify failures and accelerate root-cause analysis and remediation.

- Automate operational reporting and anomaly detection for failed tool calls, latency spikes, quality degradation, abnormal error patterns,



and unexpected AI spend.

- Establish standards for structured logging, telemetry consistency, retention policies, sensitive-data redaction, access controls, and production monitoring across ETS AI solutions.

Core Skills & Experience

- Solid hands-on experience with OpenTelemetry, distributed tracing, Azure Monitor, Application Insights, Log Analytics, and KQL.

- Practical observability experience for LLM, RAG, and multi-agent applications, including model traces, retrieval metrics, tool-call traces, quality metrics, latency, token usage, and inference-cost monitoring.

- Strong experience with dashboarding, alert engineering, incident response, production support, and operational troubleshooting; familiarity with Power BI-based reporting is valuable.

- Understanding of LangGraph and/or LangChain execution models, along with Microsoft Foundry and Copilot Studio telemetry, tracing, and evaluation data.

- Ability to collaborate closely with GenAI Engineers and Azure AI Platform Engineers to ensure telemetry is secure, standardized, actionable, and aligned with production reliability goals.

📌 AI Ops Engineer (Delhi)
🏢 IRIS SOFTWARE
📍 Delhi