About The Role

We are looking for an experienced Senior Test Engineer AI with strong expertise in software quality engineering, test automation, backend validation, and AI/ML system testing. This role requires a quality-focused engineer who can validate both deterministic software systems and probabilistic AI-powered experiences in highly scaled production environments.

The role is focused on defining and implementing test strategies for AI platform capabilities, LLM-based applications, RAG pipelines, AI agents, model integrations, backend services, and data-driven workflows. The ideal candidate should be comfortable working with Polyglotic backend systems, automation frameworks, APIs, cloud-native platforms, and AI evaluation approaches.

We require a candidate with strong analytical thinking, attention to detail, and the ability to collaborate closely with AI engineers, backend engineers, product managers, DevOps, security, and business stakeholders. The candidate should be capable of building scalable quality frameworks that support safe, reliable, measurable, and production-ready AI features.

Key Responsibilities

- Define and implement end-to-end test strategies for AI-powered applications, AI platform services, LLM integrations, RAG pipelines, AI agents, and backend systems.
- Design, build, and maintain automated test frameworks for APIs, microservices, AI workflows, model inference services, data pipelines, and event-driven systems.
- Validate AI outputs for correctness, relevance, consistency, groundedness, safety, hallucination risk, toxicity, bias, business accuracy, and user experience quality.
- Build evaluation suites using golden datasets, prompt test banks, expected response patterns, scoring rubrics, regression checks, and automated quality metrics.
- Test RAG pipelines, including document ingestion, chunking, embedding generation, vector search relevance, context retrieval, answer quality, citations, and fallback behavior.
- Perform functional, integration, regression, contract, performance, scalability, reliability, security,



and exploratory testing across AI and backend services.
- Conduct API testing for RESTful services, including authentication, authorization, request/response validation, error handling, retries, and rate limits.
- Validate LLM integration layers, including prompt templates, structured outputs, JSON response validation, guardrails, caching, cost controls, and failure handling.
- Collaborate with AI engineers, backend developers, platform teams, DevOps, product managers, and business stakeholders to define acceptance criteria and quality gates.
- Integrate automated tests into CI/CD pipelines and ensure reliable release readiness for AI and backend features.
- Perform load testing and performance benchmarking for AI inference APIs, high-scale backend services, and customer-facing AI workflows.
- Analyze production issues, logs, metrics, traces, user feedback, and AI quality regressions to identify root causes and improve test coverage.
- Prepare test plans, test cases, test reports, defect analysis, release quality dashboards, and risk assessments for AI-enabled Bachelor degree in Computer Science, Software Engineering, Information Technology, or a closely related discipline.
- Minimum of 6+ years of experience in software testing, automation testing, quality engineering, or SDET roles.
- Hands-on experience testing AI/ML systems, LLM-based applications, chatbots, recommendation systems, NLP solutions, RAG pipelines, or data-driven platforms.
- Robust automation experience using Java, Python, or Golang, with tools such as RestAssured, JUnit, TestNG, PyTest, Selenium, Playwright, Cypress, or similar.
- Strong API testing experience, including REST APIs, JSON validation, authentication,



authorization, contract testing, schema validation, and backend service validation.
- Experience testing applications in highly scaled production environments, including distributed systems, microservices, event-driven systems, and cloud-native platforms.
- Good understanding of AI/ML concepts, LLM behavior, embeddings, vector similarity, prompt engineering, context management, and AI evaluation techniques.
- Experience with cloud platforms, preferably AWS, and familiarity with Docker, Kubernetes, logs, metrics, tracing, dashboards, and production debugging.
- Strong understanding of test design techniques, risk-based testing, defect lifecycle, release governance, and quality metrics.
- Ability to identify edge cases across AI behavior, user journeys, data scenarios, integrations, security boundaries, and failure modes.
- Excellent communication, documentation, stakeholder management, and analytical skills.
- Demonstrated experience with Agile/Scrum development methodologies.

Preferred Qualifications

- Experience building AI evaluation frameworks, LLM regression suites, prompt test harnesses, or model quality dashboards.
- Familiarity with AI frameworks and tools such as LangChain, LangGraph, LlamaIndex, OpenAI APIs, Hugging Face, MLflow, or similar.
- Experience testing any vector databases such as pgvector, Pinecone, Weaviate, Milvus, Chroma, OpenSearch, or Elasticsearch.
- Experience testing event-driven systems using Kafka, RabbitMQ, SQS, SNS, Kinesis, or similar.
- Experience with security, privacy, compliance, responsible AI validation, PII handling, and guardrail testing.
- Experience with hallucination detection, toxicity testing, bias testing, fairness testing, adversarial prompt testing, and red-team style AI validation.
- Experience working with telecom platforms, SaaS products, high-scale consumer systems, or digital transformation platforms.
- Experience mentoring QA engineers or leading quality initiatives across multiple teams.

(ref:hirist.tech)

📌 Senior Test Engineer - AI/ML Testing (Bengaluru)
🏢 Information Technology
📍 Bengaluru