Responsibilities

We are seeking a highly motivated Quality Engineering (QE) Consultant Experinece in software testing, automation, and emerging AI technologies. The ideal candidate will have hands-on experience testing AI/GenAI applications , APIs , and web applications , along with strong expertise in Playwright automation and AI evaluation frameworks .

This role will focus on ensuring the quality, reliability, safety, and performance of AI-powered solutions by designing comprehensive test strategies, implementing automation frameworks, validating AI model behavior, and establishing evaluation mechanisms for large language model (LLM)-based applications.

The candidate will collaborate closely with Product Owners, Developers, Data Scientists, and AI Engineers to drive quality throughout the software development lifecycle.

AI & GenAI Testing

- Design and execute test strategies for AI/GenAI applications, including LLM-powered solutions, AI agents, copilots, and conversational systems.
- Validate AI outputs for:
- Accuracy
- Relevance
- Groundedness
- Consistency
- Hallucination detection
- Toxicity and safety compliance
- Create and maintain prompt test suites and benchmark datasets.
- Perform functional, regression, performance, and reliability testing for AI-enabled applications.
- Define and execute AI evaluation metrics and quality gates.

Automation & Quality Engineering

- Develop and maintain test automation frameworks using Playwright .
- Automate UI, API, and end-to-end business workflows.
- Create reusable automation assets and testing accelerators.
- Integrate automated tests into CI/CD pipelines.
- Support shift-left testing and quality engineering practices.

API Testing

- Design and execute API test cases using tools such as:
- Postman
- REST Assured
- Playwright API Testing
- Validate API contracts, authentication, authorization, error handling, and performance.




- Conduct integration testing across distributed systems and third-party services.

AI Evaluation & Quality Metrics

- Establish AI evaluation frameworks and quality scorecards.
- Measure model quality using evaluation techniques such as:
- Precision/Recall
- Relevancy scoring
- Semantic similarity
- Groundedness validation
- Human-in-the-loop evaluation
- Analyze AI quality trends and recommend improvements.
- Support Responsible AI and model governance requirements.

Collaboration & Stakeholder Management

- Work closely with engineering, product, and AI teams to identify quality risks early.
- Participate in requirement reviews and design discussions.
- Communicate testing progress, risks, and quality metrics to stakeholders.
- Contribute to QE best practices, standards, and reusable playbooks.

Experience

- 4–6 years of experience in Quality Engineering, Software Testing, or Test Automation.
- Hands-on experience testing AI/Generative AI applications.
- Experience with modern web application testing and API testing.

Technical Skills

- Robust expertise in Playwright automation.
- Hands-on experience with API testing and automation.
- Experience validating AI/LLM-based applications and AI agents.
- Good understanding of AI evaluation methodologies and testing techniques.
- Familiarity with:
- M365 Copilot
- OpenAI/Azure OpenAI
- Claude
- Gemini
- Agentic AI frameworks
- Knowledge of test management tools such as Azure DevOps (ADO), Jira, or similar platforms.
- Experience working with CI/CD pipelines and DevOps practices.

Preferred Skills

- Exposure to Python or TypeScript for automation and AI testing.
- Experience with AI evaluation frameworks such as:
- RAG evaluation
- Prompt evaluation
- LLM benchmarking
- Human feedback-based evaluation
- Knowledge of Responsible AI principles, AI governance, and model risk management.
- Experience testing Retrieval-Augmented Generation (RAG) systems and AI agents.

📌 Quality Engineering consultant (Bengaluru)
🏢 Artech
📍 Bengaluru