Apply on Kit Job: kitjob.in/job/43kwjm

· Remote (India)

n

· 12 PM – 8 PM IST

n

· Contract: 2 Years

n

· Onboarding: Immediate

n

About BLS360

n

BLS360 is a fast-growing cybersecurity, identity & access management (IAM), and digital transformation company , helping enterprises secure and modernize their technology landscape. Founded in 2019, BLS360 delivers scalable, secure, and customized solutions across industries including finance, healthcare, and government.

n

With strong partnerships across leading cloud and identity platforms, BLS360 focuses on identity-centric security, compliance, and next-gen digital innovation , enabling organizations to navigate complex, multi-cloud environments with confidence.

n

Role Overview

n

We’re looking for a Senior AI QA / QE Engineer to lead quality engineering efforts for cutting-edge AI-powered applications. This role is ideal for someone who thrives at the intersection of QA, automation, and AI systems , and can ensure reliability in complex, non-deterministic environments.

n

What You’ll Do

n

- Lead QA strategy and execution for AI-driven applications and systems
- Build and maintain automated test frameworks (Python, Robot Framework) across APIs, UI, backend, and data pipelines
- Integrate automated tests into CI/CD pipelines
- Perform functional, performance, and exploratory testing
- Validate AI/LLM outputs using structured and non-deterministic testing approaches
- Conduct prompt testing and evaluate LLM behavior (PromptFoo experience preferred)
- Test and validate RAG-based systems , including ground truth evaluation




- Collaborate with engineering and product teams to define quality benchmarks
- Debug complex issues, including AI/LLM-specific challenges

n

Must-Have Skills

n

- Strong experience in QA automation and test strategy
- Hands-on experience testing AI/LLM-powered applications and agents
- Expertise in non-deterministic output validation and ground truth comparison
- Experience with RAG testing and validation
- Proficiency in Python-based automation frameworks
- Solid understanding of defect analysis and system debugging
- Experience working in Agile environments with tools like Jira
- Test and validate Retrieval-Augmented Generation (RAG-based systems), including ground truth evaluation

n

Nice to Have

n

- Experience with Azure OpenAI
- Familiarity with OpenSearch (vector databases)
- Exposure to monitoring tools like Datadog

n

What Makes You a Great Fit

n

- You can identify gaps in AI response validation and improve quality rigor
- You’re comfortable working with ambiguous or partially correct AI outputs
- You bring a hands-on approach to problem-solving—not just strategy
- You can help ensure release readiness under tight timelines

n

Why Join BLS360?

n

- Work on cutting-edge AI + cybersecurity solutions
- Be part of a rapidly growing global CyberTech company
- Collaborate with global teams on impactful projects
- Chance to build impactful products, solutions, services and beyond.
- Play a key role in shaping quality standards for AI applications

Apply on Kit Job: kitjob.in/job/43kwjm

📌 AI QA / QE Engineer (Nellore)
🏢 BLS360
📍 Nellore