Key Roles & Responsibilities
• Own and drive the end-to-end test automation strategy, defining the roadmap, frameworks, tooling and standards for UI, API, backend and integration testing.
• Lead, mentor and manage the QA team, owning delivery, prioritization, test coverage and quality outcomes.
• Work closely with engineering, product, data science/ML and client service teams to clarify requirements and define acceptance criteria.
• Create and maintain test plans, test scenarios, test data, traceability and release-level quality reports.
• Triage defects, assess business and customer impact, and coordinate defect prioritization with relevant teams.
• Test AI/ML model integrations end to end, including request/response contracts, preprocessing and post-processing, timeout/retry behavior, failure handling and fallback paths.
• Build model-focused test suites using representative and edge-case datasets; assess output correctness against agreed reference labels or acceptance criteria, and track metrics such as precision/recall where applicable.
• Validate LLM-enabled workflows for groundedness, consistency, hallucination risks, prompt/input sensitivity and regression when prompts, models or versions change; use defined evaluation criteria rather than subjective checks alone.
• Check model-service latency, throughput, rate limits, availability, version compatibility and behavior under malformed, incomplete or unexpected inputs.
• Perform end-to-end, system integration and regression testing across UI, backend services, APIs, databases and third party/model services.
• Establish performance baselines and report findings with actionable recommendations for capacity and scalability improvements.
• Plan and execute performance, load, stress,



spike and endurance tests; analyze throughput, latency, concurrency, resource utilization and bottlenecks.
• Work with developers to reproduce issues, verify fixes and prevent regressions throughout the release cycle.
• Own test-suite execution in non-production environments and integrate automated checks into CI/CD pipelines where appropriate. Required Skills
• Strong experience testing web applications across UI, backend services, APIs, databases and end-to-end workflows.
• Strong QA fundamentals with substantial hands-on automation experience using Python; able to design, build, maintain and scale automation frameworks rather than only execute existing scripts.
• People leadership experience, including leading, mentoring and driving the output of a team of 3 or more QA engineers.
• Strong API testing and debugging skills, including REST APIs, response/schema validation, authentication, error handling and service-to-service integration.
• Experience with performance-testing tools such as JMeter, Locust, k6 or equivalent, and ability to interpret results and identify bottlenecks.
• Working knowledge of AI/ML model evaluation and integration testing; able to define test datasets, expected outcomes, thresholds and regression checks in collaboration with data science/ML engineers.
• Familiarity with testing LLM-based features, including prompt/version regression,



groundedness and response-quality evaluation, structured-output validation and safe failure handling.
• Comfortable validating data and service behavior using SQL, logs, metrics and traces; familiarity with CI/CD-based automated test execution is an advantage.
• Good understanding of front-end technologies and browser-based testing, including JavaScript, HTML and CSS.
• Hands-on experience building and maintaining automation frameworks using tools such as Pytest, Robot Framework, Selenium, Cypress or equivalent; experience with API automation is essential.
• Experience working in Agile/Scrum teams and participating in sprint planning, refinement, reviews and release readiness.
• A solid quality mindset and a track record of promoting automation-first, risk-based testing and continuous quality improvement.
• Explicit communication skills and the ability to work across engineering, product, QA and AI/ML teams to define and deliver quality outcomes. Behavioral Expectations
• Be excited by and have a positive outlook to navigate ambiguity.
• Passion for results and excellence.
• Team player.
• Must be able to get the job done by working collaboratively with others.
• Inquisitive and analytical mind; out-of-the-box thinking.
• Prioritize among competing opportunities, balancing consumer needs with business and product priorities, and clearly articulate the rationale behind product decisions.
• Straightforward and professional.
• Good communicator.
• Maintain high energy and motivate the team.
• A do-it-yourself orientation, consistent with the company's "roll-up-the-sleeves" culture.
• Proactive. (ref:hirist.tech)

📌 AdvaRisk - QA Lead - Automation Testing (Pune)
🏢 AdvaRisk
📍 Pune