Job Summary The Executive Manager AI Observability is a seasoned individual contributor who partners alongside Jr. AI Observability Architects to deliver high-quality, production-grade observability capabilities across the enterprise AI platform. This role brings deeper technical experience and greater independent execution capability owning delivery across one or two specialization tracks with reduced need for day-to-day direction while working as a genuine peer within the team rather than in a supervisory or mentorship capacity.

The Executive Manager is expected to be a strong, self-sufficient technical contributor who can take a complex observability requirement from design through implementation and into production operation within their assigned tracks. They bring cross-track awareness that helps the team as a whole, contribute to engineering standard discussions, and collaborate with peer architects on solving shared technical challenges all without requiring supervisory authority.

Responsibilities

Observability Platform Engineering OTEL Integration (25%)

- Design and implement OpenTelemetry (OTEL) instrumentation within one or two assigned agent frameworks or platforms including custom exporters, span enrichers, semantic convention tagging, and distributed trace context propagation with the ability to work independently from requirements through to production deployment.
- Build and maintain telemetry pipeline components (collectors, processors, exporters) that reliably route metrics, logs, traces, and semantic signals to observability backends owning the full lifecycle of assigned pipeline components including testing, deployment, and on-call support.
- Contribute to the integration of OTEL with enterprise agentic platforms (Salesforce AgentForce, ServiceNow, Microsoft Agent 365, or internal frameworks) within the assigned scope implementing instrumentation to architecture patterns established by the L11.
- Develop and maintain observability dashboards, alerting rules, and SLO/SLA definitions for assigned sub-domains validating signal quality and tuning alert thresholds to achieve low false-positive rates.
- Participate in on-call rotations and production incident response contributing to RCA documentation, runbook authoring, and post-incident improvement actions.
- Write comprehensive unit, integration, and end-to-end tests for all telemetry components owned; maintain >80% test coverage across assigned services and proactively identify gaps in existing coverage.

Safety, Security Red Teaming Observability (15%)





- Implement safety-critical signal capture within assigned telemetry pipelines building reliable instrumentation for guardrail trigger rates, policy violation events, adversarial detection flags, hallucination indicators, and trust boundary crossing alerts.
- Build observability components that support red team exercises instrumenting assigned agent systems to capture adversarial test events, behavioral deviations, and attack surface signals in a measurable, repeatable way.
- Implement secure trace handling patterns within assigned pipelines applying data masking, PII redaction, and audit-log retention configurations as specified by the security architecture.
- Contribute to the Security Observability Playbook documenting assigned instrumentation patterns, updating escalation procedures based on observed incidents, and maintaining accuracy of the playbook sections within scope.
- Monitor agent-to-agent protocol traffic (A2A, UCP, AP2) within the assigned domain for anomalous patterns flagging deviations for review in a timely manner with sufficient diagnostic context.

Responsible AI (RAI) Governance Signal Instrumentation (10%)

- Implement RAI signal collectors within assigned agent workflows building reliable pipelines that capture fairness indicators, bias detection outputs, explainability scores, and content safety classifications with validated data quality.
- Maintain RAI telemetry pipelines within scope ensuring completeness, accuracy, and timeliness of governance signals that feed into compliance dashboards, and resolving data quality issues proactively.
- Ensure all AI decision traces within the assigned domain include required governance metadata and comply with retention policies contributing to the audit-readiness of the observability platform.
- Identify and document RAI signal coverage gaps within the assigned scope reporting findings to the L11 with sufficient detail to inform remediation planning.

Quality Engineering for Agentic Solutions Post Go-Live Continuous QE (15%)

- Build and maintain quality gate components within CI/CD pipelines for assigned agent services implementing regression detection logic, performance degradation alerts, and SLA breach notifications using production observability data.
- Instrument and monitor Skill Evaluations (evals) across assigned Memory,



Skills, and MCP harness components collecting eval telemetry, tracking pass/fail trends over time, and alerting on regression thresholds with appropriate context.
- Implement continuous quality monitoring for post-go-live agentic solutions within scope tracking agent success rates, tool-call fidelity, latency distributions, and user outcome proxies against defined baselines.
- Execute structured testing of new agent capabilities using standardized eval harnesses documenting results clearly, flagging anomalies, and contributing findings to quality improvement cycles.
- Build and maintain automated quality reports and metric dashboards for assigned areas ensuring stakeholders have timely, accurate visibility into agent behavioral quality and trend direction.

Memory, Skills, MCP Harness Engineering Observability (10%)

- Instrument agent memory operations within the assigned scope building reliable monitoring of read/write latency, cache hit rates, memory staleness, and semantic drift across episodic, semantic, and working memory backends.
- Add trace instrumentation to MCP server interactions within assigned components implementing OTEL semantic tagging for tool registrations, skill invocations, context injections, and result returns.
- Capture telemetry for self-evolving harness and RL system components as assigned implementing signal capture for reward distributions, policy update events, workplace state transitions, and convergence indicators.
- Monitor eval harness execution within assigned scope building detection for flaky eval environments, setup failures, and result inconsistencies that could obscure real capability regressions.

Python Engineering Data Science Observability (10%)

- Write production-quality Python for assigned observability components custom OTEL exporters, signal aggregators, data transformation pipelines, and anomaly detection logic consistently meeting team engineering standards for code quality, testing, and documentation.
- Apply data science methods to assigned telemetry data time-series analysis, statistical threshold tuning, distribution characterization to improve signal accuracy and reduce alerting noise within the assigned domain.
- Contribute to shared Python SDK and library components implementing well-tested, documented additions that improve OTEL onboarding experience for agent developers.
- Actively participate in code reviews both receiving feedback from peers and the L11, and contributing constructive technical review of peer engineers' pull requests within areas of expertise.

📌 Executive Manager - AI Platform and Observability (Hyderabad)
🏢 PepsiCo
📍 Hyderabad