Opportunity Description AI Tester (Automation)

Our end client — a technology solutions and digital engineering organization specializing in software development, quality engineering, automation, cloud, and emerging technologies — delivers scalable and innovative technology solutions for businesses across industries. The organization focuses on helping clients accelerate digital transformation through modern engineering practices, automation, and AI-driven solutions.

They are looking for an experienced AI Tester (Automation) to design and execute testing strategies for AI/ML and LLM-powered applications. The role involves validating model accuracy, performance, reliability, and output quality, along with building automated test frameworks using tools such as Selenium, Cypress, Playwright, and Appium. The skilled will also work on AI evaluation pipelines, adversarial testing, hallucination and prompt-injection testing, edge-case validation, and data quality to ensure robust and production-ready AI solutions.

Roles & Responsibilities

- Design & execute AI/ML model test strategies accuracy, performance, consistency &

reliability
- Test LLM-powered features for hallucinations, prompt injection, bias & output quality
- Build & maintain automated test frameworks — Selenium, Cypress, Playwright, Appium
- Develop AI evaluation pipelines — test LLM responses for relevance, coherence &

toxicity




- Perform adversarial testing & edge case generation — probe AI with boundary inputs
- Validate data quality & training datasets — identify noise, bias & inconsistencies
- Conduct regression testing to ensure model updates don't degrade AI performance
- Monitor AI model drift & flag performance degradation in production
- Execute API & integration testing for AI services — Postman or RestAssured
- Integrate AI test suites into CI/CD pipelines — Jenkins, GitHub Actions
- Log & track AI-specific bugs — hallucinations, false positives, model failures
- Collaborate with Data Science & ML engineers on quality benchmarks
- Participate in agile ceremonies as the AI quality advocate

Required Skills

- A curious, detail-obsessed tester with 2+ years in QA automation + AI testing
- Strong in Test Automation — Selenium / Cypress / Playwright / Appium Must Have
- Hands-on with testing AI/ML models or LLM-powered features
- Familiar with prompt engineering & how prompts impact AI output quality
- Proficient in API testing — Postman, RestAssured
- Comfortable scripting in Python or JavaScript
- Understanding of ML concepts — accuracy, precision, recall, F1 score
- Familiar with AI testing tools — Pytest, MLflow, Great Expectations
- Hands-on with CI/CD — Jenkins, GitHub Actions, or GitLab CI
- Self-starter with ownership mindset & proactive communication
- Ready to work onsite in Noida
- Excellent communication and collaboration skills.

📌 AI Tester (Automation) (Greater Noida)
🏢 hiringhood
📍 Greater Noida