Position Description:

We are seeking an Automation Quality Engineer with solid hands-on experience in Java and/or Python automation frameworks and practical exposure to AI-driven testing. The ideal candidate will play a key role in validating intelligent applications, ensuring the quality, reliability, and performance of AI-enabled systems. This role involves testing across traditional automation frameworks as well as advanced AI workflows, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), Vector Databases, and OpenAI APIs.

Your future duties and responsibilities

- Design, develop, and maintain automation test frameworks using Java and/or Python.
- Implement UI, API, and backend automation using tools such as Selenium, Playwright, REST Assured, PyTest, or similar.
- Validate LLM-based applications, ensuring prompt accuracy, response quality, hallucination detection, and consistency.
- Test and evaluate RAG pipelines, including document ingestion, embeddings, retrieval accuracy, and response relevance.
- Work with Vector Databases (FAISS, Pinecone, Chroma, Weaviate) to validate indexing, retrieval, and performance.
- Integrate and test applications using OpenAI APIs or similar LLM providers.
- Leverage LangChain (or equivalent frameworks) for testing chains, agents, tools, and workflows.
- Create automated test cases for AI workflows, model outputs, prompt versions, and inference pipelines.




- Implement data-driven and AI-assisted testing strategies.
- Collaborate with Data Scientists, Developers, and Product teams to define quality metrics for AI systems.
- Participate in CI/CD pipelines to enable continuous testing and enforce quality gates.
- Ensure compliance with security, performance, and scalability standards.

Required qualifications to be successful in this role

- Strong programming skills in Python.
- Hands-on experience with automation testing frameworks (Selenium, Playwright, REST Assured, PyTest, etc.).
- Practical experience with LLMs (OpenAI GPT, Azure OpenAI, or similar).
- Solid understanding and exposure to RAG architectures.
- Experience working with Vector Databases for embeddings and semantic search.
- Hands-on usage of OpenAI APIs (chat/completions, embeddings, etc.).
- Experience with LangChain for prompt chaining, memory, and tool usage.
- Strong knowledge of API testing, SQL, and data validation.
- Familiarity with CI/CD tools (Jenkins, GitHub Actions, Azure DevOps).
- Good understanding of SDLC, STLC, and Agile methodologies.

Good-to-Have Skills

- Certification in LLMs and/or Python.
- Exposure to AI-assisted testing strategies and emerging AI QA tools.
- Experience with performance testing of AI-enabled systems.
- Knowledge of cloud-based AI platforms and orchestration frameworks

📌 AI Automation (Bengaluru)
🏢 CGI
📍 Bengaluru