AI Ops Engineer

ETS AI | GenAI Observability & Operations

Role Purpose
Own post-deployment observability, reliability, and operational health for ETS AI agents across Copilot Studio and pro-code Azure applications.
Replace manual, Power BI-centered monitoring with automated dashboards, proactive alerts, and actionable production telemetry for enterprise AI workloads.

Key Responsibilities
Instrument agents, workflows, and supporting services using OpenTelemetry and distributed tracing across user requests, model calls, retrieval steps, tool execution, and agent workflow stages.
Build operational dashboards using Azure Monitor, Application Insights, Log Analytics, and KQL; integrate Copilot Studio and Microsoft Foundry telemetry where available.
Define and operationalize alerts, thresholds, SLIs, and SLOs covering availability, latency, error rates, task completion, retrieval quality, token consumption, and inference cost.
Correlate agent runs, distributed traces, evaluation results, logs, and production incidents to identify failures and accelerate root-cause analysis and remediation.
Automate operational reporting and anomaly detection for failed tool calls, latency spikes, quality degradation, abnormal error patterns,



and unexpected AI spend.
Establish standards for structured logging, telemetry consistency, retention policies, sensitive-data redaction, access controls, and production monitoring across ETS AI solutions.

Core Skills & Experience
Solid hands-on experience with OpenTelemetry, distributed tracing, Azure Monitor, Application Insights, Log Analytics, and KQL.
Practical observability experience for LLM, RAG, and multi-agent applications, including model traces, retrieval metrics, tool-call traces, quality metrics, latency, token usage, and inference-cost monitoring.
Solid experience with dashboarding, alert engineering, incident response, production support, and operational troubleshooting; familiarity with Power BI-based reporting is valuable.
Understanding of LangGraph and/or LangChain execution models, along with Microsoft Foundry and Copilot Studio telemetry, tracing, and evaluation data.
Ability to collaborate closely with GenAI Engineers and Azure AI Platform Engineers to ensure telemetry is secure, standardized, actionable, and aligned with production reliability goals.

📌 Ai Ops Engineer Noida
🏢 IRIS SOFTWARE
📍 Noida