Sr. AI QA Engineer - Immediate joiners required 2 days WFO - Hinjewadi Phase 3

Who are we

Fulcrum Digital is an agile and next-generation digital accelerating company providing digital transformation and technology services right from ideation to implementation. These services have applicability across a variety of industries, including banking & financial services, insurance, retail, higher education, food, healthcare, and manufacturing.

Key Responsibilities
- Design and execute test cases for LLM agents, RAG pipelines, agentic workflows, and AI-assisted decision tools
- Validate AI outputs against ground truth using structured accuracy scoring (NAICS, risk exposure flags, hazard group mapping)
- Detect hallucinations, reasoning gaps, source fabrication, and misattributions in model-generated content
- Run multi-model comparative testing across GPT, Claude, Gemini, and Perplexity evaluating accuracy, latency, and output completeness
- Test prompt versions iteratively and track accuracy changes across prompt cycles
- Validate citation accuracy, document ingestion pipelines,



and cross-document context handling
- Design edge case and negative tests for AI-specific failure modes content filter triggers, tool call limits, missing documents, and incomplete synthesis
- Perform regression testing after model upgrades, prompt changes, and backend fixes, and maintain structured QA sign-off in JIRA

What Makes This Role Different from Traditional QA
- You evaluate whether an AI is reasoning correctly not just whether the UI behaves as expected
- You build evaluation rubrics for non-deterministic outputs and apply LLM-as-a-Judge techniques to score quality at scale
- You treat every model or prompt change as a potential accuracy regression, not just a functional one
- You understand that in live AI systems, a passing test today does not guarantee a passing test tomorrow

Required Skillsets
- 78 years of QA experience with minimum 2 years in Generative AI / LLM-based project

📌 Senior AI QA engineer (Pune)
🏢 Fulcrum Worldwide Software
📍 Pune