Sensitivity Label: General AI Agent Solution Specialist Job Description This role supports both AI-enabled and human-assisted customer interactions by configuring no-code AI workflows, monitoring performance, and analyzing customer interaction data to drive continuous improvement, operational efficiency, and positive customer outcomes within a contact center environment. What You'll Do AI Agent Journey Design & Configuration - Design, build, and continuously evolve no-code AI agent journeys, including conversation flows, decision logic, and end-to-end user experiences. - Configure intents, prompts, business rules, and escalation paths using intuitive tools, enabling scalable content management without direct coding. - Collaborate with product owners, engineers, and business stakeholders to translate requirements into scalable, no-code agent experiences. - Ensure all agent journeys align with governance, security, and responsible AI practices across the agent development lifecycle. Performance Monitoring & Analytics - Monitor and benchmark AI agent performance across journeys (accuracy, containment, resolution rate, user satisfaction), applying simulation-driven thinking to real-world scenarios. - Analyze interaction logs and journey analytics to identify drop-offs, failure patterns, and optimization opportunities, feeding insights into continuous improvement loops. - Design and maintain advanced search and query frameworks (speech and text, pattern-based logic) to enable automated analysis and topic identification in customer interactions. - Build and maintain advanced speech and text queries (including pattern-based logic) to monitor both human and AI-assisted interactions. - Generate actionable insights from interaction data and communicate findings to CX stakeholders, supporting data-driven decision making. AI Agent Quality Management - Own the end-to-end quality management framework for AI agents - defining quality standards, evaluation criteria, scoring rubrics, and pass/fail thresholds across all deployed agent journeys. - Conduct systematic conversation reviews and audits of AI agent interactions, scoring responses for accuracy, tone, compliance, escalation appropriateness, and resolution quality. Sensitivity Label: General - Develop and maintain a structured QA scorecard tailored to AI agent interactions, incorporating dimensions such as intent recognition accuracy, hallucination detection,



knowledge retrieval relevance, and conversation coherence. - Identify recurring quality defects, failure modes, and edge cases through interaction sampling and trend analysis - distinguishing between prompt-level, knowledge-level, and integration-level root causes. - Establish and run calibration sessions with cross-functional stakeholders (product, engineering, CX) to ensure consistent quality evaluation standards across agent deployments. - Track quality metrics over time (QA pass rate, critical defect rate, regression frequency) and report trends to leadership with clear improvement recommendations. Training Feedback & AI Engineer Collaboration - Translate quality findings into structured, actionable feedback for AI Agent Engineers - providing specific examples, annotated conversation logs, and clear descriptions of expected vs. actual agent behavior. - Maintain a prioritized defect and improvement backlog informed by QA findings, categorized by severity, frequency, and customer impact - collaborating with engineers to drive resolution. - Participate in regular feedback loops with engineering, reviewing prompt refinements, knowledge base updates, and guardrail adjustments to validate that quality issues are resolved without introducing regressions. - Develop and curate a library of gold-standard conversation examples and failure-case annotations that serve as training references for prompt tuning, knowledge curation, and agent behavior calibration. - Contribute to the design of automated evaluation pipelines by defining test scenarios, expected outputs, and quality assertions that engineers can integrate into CI/CD workflows. - Support the creation of regression test suites by documenting resolved defects as repeatable test cases, ensuring fixed issues do not resurface across agent updates. - Partner with engineers during post-deployment reviews to assess whether agent updates have improved quality metrics, using before-and-after analysis of QA scores and interaction outcomes. Testing,



Experimentation & Continuous Improvement - Test and validate AI agent behavior through structured experimentation (A/B testing, edge case validation), ensuring quality, compliance, and responsible AI standards. - Investigate incidents and unexpected agent behavior, conducting root-cause analysis in non-deterministic AI systems. - Contribute to the evolution of self-improving, generative agent systems by leveraging real-world interactions and feedback loops. Sensitivity Label: General What You'll Bring - Passion for working at the frontier of AI products, especially in generative AI and agent-based systems. - Language proficiency in English, French, Spanish (written and spoken). - High ownership mindset with the ability to operate autonomously, navigate ambiguity, and drive meaningful outcomes. - Solid analytical and problem-solving skills, with the ability to interpret complex interaction data and translate insights into action. - Excellent verbal and written communication skills, with the ability to clearly convey findings to both technical and non-technical stakeholders. - Ability to manage multiple priorities independently in a deadline-driven environment, while collaborating effectively across cross-functional teams. - Strong planning, organizational, and time-management skills. - A quality-first mindset - methodical attention to detail in reviewing AI agent outputs, with the discipline to maintain consistent evaluation standards across high volumes of interactions. - Comfort operating in the feedback loop between quality evaluation and engineering execution - able to articulate what's wrong, why it matters, and what good looks like. Nice to Have - Experience with quality assurance and training in a contact center or BPO environment. - Experience supporting contact center technologies, particularly speech analytics and AI-driven interaction platforms. - Experience building AI-powered products, particularly with LLMs, conversational AI, or autonomous agents. - Hands-on experience with AI agent evaluation frameworks, including conversation scoring, automated testing, and regression analysis. - Familiarity with prompt engineering and knowledge base curation as levers for improving agent quality. - Experience creating QA rubrics or scorecards for conversational AI or chatbot deployments.

📌 Software Development Senior Analyst (Bengaluru)
🏢 NTT America
📍 Bengaluru