Key Roles & Responsibilities

- Own and drive the end-to-end test automation strategy, defining the roadmap, frameworks, tooling and standards for UI, API, backend and integration testing.
- Lead, mentor and manage the QA team, owning delivery, prioritization, test coverage and quality outcomes.
- Work closely with engineering, product, data science/ML and client service teams to clarify requirements and define acceptance criteria.
- Create and maintain test plans, test scenarios, test data, traceability and release-level quality reports.
- Triage defects, assess business and customer impact, and coordinate defect prioritization with relevant teams.
- Test AI/ML model integrations end to end, including request/response contracts, preprocessing and post-processing, timeout/retry behavior, failure handling and fallback paths.
- Build model-focused test suites using representative and edge-case datasets; assess output correctness against agreed reference labels or acceptance criteria, and track metrics such as precision/recall where applicable.
- Validate LLM-enabled workflows for groundedness, consistency, hallucination risks, prompt/input sensitivity and regression when prompts, models or versions change; use defined evaluation criteria rather than subjective checks alone.
- Check model-service latency, throughput, rate limits, availability, version compatibility and behavior under malformed, incomplete or unexpected inputs.
- Perform end-to-end, system integration and regression testing across UI, backend services, APIs, databases and third party/model services.
- Establish performance baselines and report findings with actionable recommendations for capacity and scalability improvements.
- Plan and execute performance, load, stress,



spike and endurance tests; analyze throughput, latency, concurrency, resource utilization and bottlenecks.
- Work with developers to reproduce issues, verify fixes and prevent regressions throughout the release cycle.
- Own test-suite execution in non-production environments and integrate automated checks into CI/CD pipelines where appropriate.

Required Skills

- Strong experience testing web applications across UI, backend services, APIs, databases and end-to-end workflows.
- Strong QA fundamentals with substantial hands-on automation experience using Python; able to design, build, maintain and scale automation frameworks rather than only execute existing scripts.
- People leadership experience, including leading, mentoring and driving the output of a team of 3 or more QA engineers.
- Strong API testing and debugging skills, including REST APIs, response/schema validation, authentication, error handling and service-to-service integration.
- Experience with performance-testing tools such as JMeter, Locust, k6 or equivalent, and ability to interpret results and identify bottlenecks.
- Working knowledge of AI/ML model evaluation and integration testing; able to define test datasets, expected outcomes, thresholds and regression checks in collaboration with data science/ML engineers.
- Familiarity with testing LLM-based features,



including prompt/version regression, groundedness and response-quality evaluation, structured-output validation and safe failure handling.
- Comfortable validating data and service behavior using SQL, logs, metrics and traces; familiarity with CI/CD-based automated test execution is an advantage.
- Good understanding of front-end technologies and browser-based testing, including JavaScript, HTML and CSS.
- Hands-on experience building and maintaining automation frameworks using tools such as Pytest, Robot Framework, Selenium, Cypress or equivalent; experience with API automation is essential.

- Experience working in Agile/Scrum teams and participating in sprint planning, refinement, reviews and release readiness.
- A strong quality mindset and a track record of promoting automation-first, risk-based testing and continuous quality improvement.
- Clear communication skills and the ability to work across engineering, product, QA and AI/ML teams to define and deliver quality outcomes.

Behavioral Expectations

- Be excited by and have a positive outlook to navigate ambiguity.
- Passion for results and excellence.
- Team player.
- Must be able to get the job done by working collaboratively with others.
- Inquisitive and analytical mind; out-of-the-box thinking.
- Prioritize among competing opportunities, balancing consumer needs with business and product priorities, and clearly articulate the rationale behind product decisions.
- Straightforward and qualified.
- Good communicator.
- Maintain high energy and motivate the team.
- A do-it-yourself orientation, consistent with the company's "roll-up-the-sleeves" culture.
- Proactive.

(ref:hirist.tech)

📌 AdvaRisk - QA Lead - Automation Testing (Pune)
🏢 AdvaRisk
📍 Pune