AI Quality Engineer – LLM Evaluation & Mobile Automation

ROLE PURPOSE

We are looking for an AI Quality Engineer to build and scale quality assurance for AI-powered features, including LLM-based assistants, search, recommendations, and document understanding. The role combines automated LLM evaluation, mobile and web test automation, API validation, and performance testing. The engineer will own automation frameworks, integrate quality checks into CI/CD pipelines, and proactively identify model regressions, data issues, and reliability risks.

KEY RESPONSIBILITIES

Build and maintain automated LLM evaluation suites (DeepEval, Ragas, promptfoo, LangSmith).

Evaluate AI responses for accuracy, groundedness, relevance, and refusal behaviour.

Maintain golden datasets and track hallucination rates to catch regressions pre-release.

Validate multi-turn conversational workflows, tool calling, and failure recovery.

Test AI safety and guardrails: prompt injection, jailbreaks, PII leakage, blocked topics.

Monitor AI quality, latency, cost, and drift; convert production failures into regression tests.





Validate AI pipeline data using SQL/AWS Athena and verify data protection controls.

Own the Appium automation framework for Android/iOS, including BLE and Wi-Fi provisioning.

Build web/API automation (Playwright, Selenium, Postman) and run load/performance tests.

MUST-HAVE EXPERIENCE

4-6 years in test automation with hands-on ownership of frameworks and test strategy.

Practical experience with an LLM evaluation framework (DeepEval, Ragas, promptfoo, LangSmith).

Solid Python skills with solid OOP concepts and reusable test frameworks.

Hands-on Appium experience for Android and/or iOS automation.

Understanding of non-deterministic AI outputs and evaluation beyond simple pass/fail.

Experience with Playwright or Selenium for web automation.

Proficiency in REST API testing, Swagger/OpenAPI, Postman, and SQL for data validation.

Working knowledge of Git, CI/CD pipelines, and continuous testing workflows.

Working knowledge of AWS (S3, Lambda, CloudWatch, IAM) and experience with AWS Device Farm.

📌 Specialist - AI Quality Engineer – LLM Evaluation & Mobile Automation (Pune)
🏢 Atomberg technologies
📍 Pune