20 Nov
|
Qentelli
|
Hyderabad
20 Nov
Qentelli
Hyderabad
Job Summary
We are seeking an experienced AI Quality Engineering / AI Test Lead with expertise in testing AI/ML, LLM-based, and Agentic AI systems. The ideal candidate will be responsible for validating the reliability, performance, scalability, safety, and compliance of AI-driven applications and multi-agent architectures. This role involves designing robust test frameworks, evaluating LLM outputs, implementing AI guardrails, and ensuring production-grade quality standards across distributed AI ecosystems.
Roles and Responsibilities
- Design, develop, and execute comprehensive test strategies for AI/ML, LLM, and agentic AI applications.
- Build and maintain automated test harnesses, simulators, fixtures, and validation frameworks using Python.
- Evaluate LLM performance using metrics such as semantic similarity, BLEU, ROUGE, BERTScore, and exact/soft matching techniques.
- Validate AI guardrails, prompt reliability, content moderation policies, and model safety mechanisms.
- Perform end-to-end testing of multi-agent workflows, orchestration frameworks, and AI decision-making systems.
- Conduct distributed systems testing, including latency profiling, resilience validation, failure recovery, and chaos engineering.
- Implement and optimize CI/CD pipelines for AI testing and deployment validation.
- Leverage observability tools to monitor AI application behavior, performance, and production readiness.
- Collaborate with Engineering, Data Science, Operations, and Product teams to define quality standards and release criteria.
- Ensure compliance with privacy, security, and regulatory requirements related to AI systems, including PII handling and responsible AI practices.
- Drive continuous improvements in AI testing methodologies, automation frameworks, and evaluation pipelines.
Mandatory Skills
- 6+ years of experience in AI Development with at least 2+ years focused on AI/ML, Generative AI,
or LLM-based systems.
- Solid expertise in testing Agentic AI and Multi-Agent architectures.
- Advanced programming skills in Python, including experience building test automation frameworks, simulators, and fixtures.
- Hands-on experience with LLM evaluation methodologies such as BLEU, ROUGE, BERTScore, semantic similarity, exact/soft matching, prompt testing, and guardrails validation.
- Expertise in distributed systems testing, latency profiling, resiliency testing, chaos engineering, retries, circuit breakers, and message queue validation.
- Experience with AI orchestration frameworks such as LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, or Azure OpenAI.
- Strong knowledge of CI/CD pipelines using GitHub Actions, Azure DevOps, or similar platforms.
- Experience with observability and monitoring tools including OpenTelemetry, Prometheus, Grafana, and Datadog.
- Understanding of feature flags, canary deployments, and production release validation strategies.
- Good understanding of AI security, privacy, governance, compliance, and model safety requirements.
- Strong communication, stakeholder management, and cross-functional collaboration skills.
Preferred Skills
- Experience with multi-agent simulators, agent graph validation, and tooling latency emulation.
- Knowledge of MLOps practices including model versioning, dataset management, evaluation pipelines, and model lifecycle management.
- Experience implementing A/B testing and experimentation frameworks for LLM applications.
- Strong background in AWS cloud services, serverless technologies, containerization, and event-driven architectures.
- Experience managing production AI workloads with ownership of cost optimization, latency reduction, scalability, and SLA adherence.
- Familiarity with Generative AI evaluation frameworks and Responsible AI best practices.
- Exposure to Kubernetes, Docker, and cloud-native application testing.
📌 AI Engineer (Hyderabad)
🏢 Qentelli
📍 Hyderabad