Role & responsibilities

The Domain SME / QA Evaluation Designer is responsible for crafting business-realistic natural language questions and verifiable ground-truth answers across the five engagement domains (Device Management, IT Ops/Incidents, CRM/Sales, E-Commerce, Finance/GL). Two positions are available. Each resource will own 23 domains. This role requires the rare combination of genuine business domain knowledge and robust SQL proficiency the person must be able to write questions that a real business analyst would ask and verify the answers against a relational dataset.

Preferred candidate profile

QA Question Design (Primary Responsibility)

- Author natural-language questions across 5 question tiers: single-table aggregations, INNER JOINs, LEFT OUTER JOINs, 3-table joins, GROUP BY + HAVING
- Ensure every question reads as a genuine business-analyst question — not SQL terminology disguised in prose
- Write the derivation_rationale for each QA pair: step-by-step explanation of how the answer is computed from the dataset




- Author the judge_reference answer for each pair — the model-level anchor used for LLM-as-Judge calibration
- Validate that each question has exactly one deterministic answer from the golden dataset; no ambiguity in expected output

Domain Realism & Schema Review

- Review ER diagrams and table designs for domain realism: column names, data ranges, entity relationships must reflect real-world patterns
- Propose domain-specific edge cases: e.g. devices with no policy assignment (LEFT JOIN NULL), incidents with no SLA record
- Review the imperfection injection plan: advise which fields should have NULLs and what duplicate patterns are realistic for the domain

LLM-as-Judge Calibration

- Provide 10+ human-graded anchor QA examples per domain for LLM-as-Judge calibration
- Score 10–15 sample Pulse API responses per domain against the 4-dimension rubric to establish the human calibration baseline
- Review and validate the LLM-as-Judge Scoring Rubric PDF for domain accuracy before delivery to customer

📌 Domain SME Q A Evaluation Designer Senior (Telangana)
🏢 Codehive Labs Hyderabad
📍 Telangana