Role Overview /n We are looking for an experienced QA Manager to lead our quality engineering function, with a strong focus on test automation, evaluation of AI/LLM-based systems, and team leadership. The ideal candidate combines deep hands-on technical expertise with proven experience managing and mentoring QA teams. /n Key Details /n /n

- Position: QA Manager – Automation

/n

- Total Experience: 10 to 17 years

/n

- Team Management Experience: Minimum 2 years (leading QA/automation teams)

/n

- Employment Type: Full-time

/n /n Roles & Responsibilities /n /n

- Lead, mentor, and grow a team of QA/automation engineers, driving their technical and professional development.

/n

- Define and own the overall test automation strategy, framework architecture, and quality roadmap for the product.

/n

- Design, build, and maintain scalable automation frameworks using Playwright for web application testing.

/n

- Drive evaluation (eval) strategies for AI/LLM-based features — designing test harnesses, golden datasets, scoring rubrics, and regression suites for model outputs.

/n

- Partner with data science/ML engineering teams to define quality benchmarks for LLM-powered and AI-driven systems, including hallucination checks, output consistency, and prompt regression testing.

/n

- Write and review robust, maintainable automation code in Java, integrating with CI/CD pipelines.

/n

- Establish QA processes, best practices, and metrics (coverage, defect leakage, automation ROI) across the team.

/n

- Collaborate closely with Product, Engineering, and Data Science stakeholders to define acceptance criteria and quality gates, including for AI-driven features.

/n

- Own test planning, risk assessment, release sign-off, and reporting to leadership.





/n

- Stay current with emerging tools and techniques in AI/LLM testing and evaluation methodologies.

/n /n Required Skills & Experience /n /n

- 10–17 years of total QA/testing experience, including automation-focused roles.

/n

- Minimum 2 years of experience managing/leading a QA or automation team.

/n

- Strong hands-on experience with Playwright for automated UI/end-to-end testing.

/n

- Solid Java programming skills for building and maintaining test frameworks.

/n

- Demonstrated experience testing AI/LLM-based systems — prompt testing, model output validation, regression testing for generative AI features.

/n

- Practical knowledge of eval skills — designing evaluation frameworks, benchmarks, scoring criteria, and golden datasets for AI systems.

/n

- Working knowledge of LLM concepts (prompting, context windows, hallucinations, RAG, fine-tuning basics) sufficient to design meaningful test/eval strategies.

/n

- Experience with CI/CD integration for automated test suites (Jenkins, GitHub Actions, or similar).

/n

- Solid understanding of QA methodologies: functional, regression, integration, performance, and exploratory testing.

/n

- Excellent communication and stakeholder management skills.

/n /n Good to Have /n /n

- Experience with additional automation tools (Selenium, Cypress, etc.).

/n

- Exposure to API testing tools (Postman, REST Assured).

/n

- Familiarity with cloud platforms (AWS/Azure/GCP) and containerized test environments.

/n

- Prior experience in a product-based or SaaS company testing AI-powered features.

/n /n Qualifications /n /n

- Bachelor's/Master's degree in Computer Science, Engineering, or a related field (or equivalent practical experience).

/n

📌 Quality Assurance Manager (Bengaluru)
🏢 Tekion
📍 Bengaluru