QA Automation Engineer - AI & Application Testing

Experience : 3 Years

Location : Remote

Employment Type : Full-time

About LegacyLeap :

LegacyLeap is a fast-growing technology startup building AI-powered solutions for enterprise application modernization.

We're working at the intersection of GenAI, Agentic AI, data intelligence, and software engineering to solve complex real-world problems around legacy enterprise systems.

This is an opportunity to join an early-stage, engineering-driven team where you won't just execute predefined tasks - you'll have the opportunity to build, experiment, challenge existing approaches, and directly influence the quality of the product.

If you enjoy working on challenging engineering problems and want to grow with a startup that is building something meaningful with AI, we'd love to hear from you.

About the Role :

We are looking for a QA Automation Engineer with strong core QA fundamentals, hands-on coding skills in Python or Java, and experience in automation, API testing and CI/CD.

The ideal candidate has exposure to AI/GenAI/LLM or Agentic AI application testing and brings a strong QA mindset - someone who naturally looks for gaps, edge cases and opportunities to improve the product and testing process.

What You'll Do :

- Design and develop scalable test automation frameworks for web, mobile, API and modern applications.
- Perform functional, integration, regression and end-to-end testing.
- Build automation using Python or Java and frameworks such as PyTest, Playwright, Selenium, TestNG or similar.
- Integrate automated testing into CI/CD pipelines.
- Test AI/GenAI/LLM-powered applications and Agentic AI workflows.
- Evaluate AI outputs for correctness, relevance, consistency and quality.
- Identify AI-specific risks including hallucinations, prompt injection, jailbreaks and unexpected behaviour.
- Perform API and backend/data validation.




- Work closely with Product and Engineering teams to identify quality risks early.
- Continuously improve test coverage, automation, processes and overall product quality.
- Quickly adapt to new technologies, products and business domains.

Must-Have Skills :

- 3 years of QA / Software Testing / Automation experience
- Strong understanding of Core QA concepts
- Good programming skills in Python or Java
- Strong problem-solving and logical reasoning skills
- Hands-on automation experience with PyTest, Playwright, Selenium, TestNG or similar
- API testing/automation experience
- CI/CD experience
- SQL/backend validation
- Experience with AI/GenAI/LLM application testing is highly preferred
- Experience with AI evaluation / LLM evaluation / RAG / AI Agent testing is a strong plus

Positive to Have :

- Mobile application testing
- Linux / Unix and shell scripting
- RAGAS, DeepEval, LangSmith or similar AI evaluation tools
- Conversational AI / chatbot testing
- Agentic AI testing
- Experience across multiple projects, products or domains
- Product-based company experience

Who Will Thrive Here?

Someone who doesn't simply ask: "What should I test?" - but asks: "What could go wrong? What are we missing? How can we make this better?"

We value strong engineering fundamentals, curiosity, ownership, adaptability and a continuous-improvement mindset.

Why Join LegacyLeap?

- Growing startup - Be part of the journey from an early stage.
- Work with AI - Get hands-on exposure to GenAI, Agentic AI and intelligent enterprise systems.
- Build, don't just maintain - Your work will directly shape products and engineering practices.
- High ownership - Opportunity to take responsibility and make an impact.
- Learn & grow - Work alongside experienced engineers on challenging real-world problems.
- Remote - Work with a distributed, collaborative team.

If you're excited about QA, automation and the future of AI-powered software, come build it with us.

📌 Quality Assurance Automation Engineer - AI & Application Testing (India)
🏢 Legacyleap
📍 India