We are looking for a Senior Automation QA Engineer to own end-to-end test automation across API and UI layers for our AI-native platform, including agent workflow validation.
Key Responsibilities
Design and execute test strategies for API and UI layers using Pytest and Playwright (Angular).
Own CI/CD quality gates in Azure DevOps, ensuring reliable build promotion and release readiness.
Build and maintain coverage for agent workflow testing, including non-determinism, agent graph validation, and confidence score assertions.
Develop reusable automation frameworks and data setup utilities backed by PostgreSQL.
Contribute to LLM evaluation and observability workflows using Langfuse or similar tooling.
Required Qualifications
6+ years of experience in QA automation.
Robust hands-on expertise with Pytest and Playwright, with proven depth in both API and UI automation.
Solid proficiency in Python and experience working with PostgreSQL for test data and validation.
Hands-on experience with Azure DevOps CI/CD pipelines and quality gates.
Awareness of LLM/agent testing challenges (non-determinism, agent graph validation, confidence score assertions); exposure to Langfuse or LLM evaluation frameworks is a plus and can be built on the job.