Key Roles & Responsibilities : - Own and drive the end-to-end test automation strategy, defining the roadmap, frameworks, tooling and standards for UI, API, backend and integration testing.- Lead, mentor and manage the QA team, owning delivery, prioritization, test coverage and quality outcomes.- Work closely with engineering, product, data science/ML and client service teams to clarify requirements and define acceptance criteria.- Create and maintain test plans, test scenarios, test data, traceability and release-level quality reports.- Triage defects, assess business and customer impact, and coordinate defect prioritization with relevant teams.- Test AI/ML model integrations end to end, including request/response contracts, preprocessing and post-processing, timeout/retry behavior, failure handling and fallback paths.- Build model-focused test suites using representative and edge-case datasets; assess output correctness against agreed reference labels or acceptance criteria, and track metrics such as precision/recall where applicable.- Validate LLM-enabled workflows for groundedness, consistency, hallucination risks, prompt/input sensitivity and regression when prompts, models or versions change; use defined evaluation criteria rather than subjective checks alone.- Check model-service latency, throughput, rate limits, availability, version compatibility and behavior under malformed, incomplete or unexpected inputs.- Perform end-to-end, system integration and regression testing across UI, backend services, APIs, databases and third party/model services.- Establish performance baselines and report findings with actionable recommendations for capacity and scalability improvements.- Plan and execute performance, load, stress,



spike and endurance tests; analyze throughput, latency, concurrency, resource utilization and bottlenecks.- Work with developers to reproduce issues, verify fixes and prevent regressions throughout the release cycle.- Own test-suite execution in non-production environments and integrate automated checks into CI/CD pipelines where appropriate.Required Skills : - Strong experience testing web applications across UI, backend services, APIs, databases and end-to-end workflows.- Strong QA fundamentals with substantial hands-on automation experience using Python; able to design, build, maintain and scale automation frameworks rather than only execute existing scripts.- People leadership experience, including leading, mentoring and driving the output of a team of 3 or more QA engineers.- Strong API testing and debugging skills, including REST APIs, response/schema validation, authentication, error handling and service-to-service integration.- Experience with performance-testing tools such as JMeter, Locust, k6 or equivalent, and ability to interpret results and identify bottlenecks.- Working knowledge of AI/ML model evaluation and integration testing; able to define test datasets, expected outcomes, thresholds and regression checks in collaboration with data science/ML engineers.- Familiarity with testing LLM-based features,



including prompt/version regression, groundedness and response-quality evaluation, structured-output validation and safe failure handling.- Comfortable validating data and service behavior using SQL, logs, metrics and traces; familiarity with CI/CD-based automated test execution is an advantage.- Positive understanding of front-end technologies and browser-based testing, including JavaScript, HTML and CSS.- Hands-on experience building and maintaining automation frameworks using tools such as Pytest, Robot Framework, Selenium, Cypress or equivalent; experience with API automation is essential.- Experience working in Agile/Scrum teams and participating in sprint planning, refinement, reviews and release readiness.- A strong quality mindset and a track record of promoting automation-first, risk-based testing and continuous quality improvement.- Clear communication skills and the ability to work across engineering, product, QA and AI/ML teams to define and deliver quality outcomes.Behavioral Expectations : - Be excited by and have a positive outlook to navigate ambiguity.- Passion for results and excellence.- Team player.- Must be able to get the job done by working collaboratively with others.- Inquisitive and analytical mind; out-of-the-box thinking.- Prioritize among competing opportunities, balancing consumer needs with business and product priorities, and clearly articulate the rationale behind product decisions.- Straightforward and professional.- Good communicator.- Maintain high energy and motivate the team.- A do-it-yourself orientation, consistent with the company's "roll-up-the-sleeves" culture.- Proactive. (ref:hirist.tech)

📌 AdvaRisk - QA Lead - Automation Testing (Pune)
🏢 AdvaRisk
📍 Pune