AI Engineer: Agents & Evaluation (Gurugram)

AI Engineer: Agents & Evaluation (Gurugram)

19 Nov
|
WOW Softech
|
Gurugram

19 Nov

WOW Softech

Gurugram

Duties & Accountabilities

- Agent engineering: design and build the initial BA, Dev, and QA agents: prompt architecture, context assembly, tool use, and schema-validated JSON outputs, with prompts and skills packs versioned in git and treated as core IP.
- Retrieval & context: build the codebase retrieval layer (syntax-aware chunking with tree-sitter, embeddings in pgvector) and the context-pack system: project glossary, org conventions, constraint register, and exemplar retrieval.
- Evaluation harness: own the golden dataset (2530 expert-verified items) and the harness that runs nightly in CI: per-agent scores against published acceptance bars, trend monitoring, and regression alerts that block release.
- Failure modes: design for the ways agents fail: schema-repair loops, reject-loop re-invocation with reviewer comments injected, retrieval-verified symbols to guard against hallucinated APIs, and honest gap declaration over coverage inflation.
- Confidence & quality: implement and tune the confidence-scoring model that routes low-confidence outputs to senior review, and use every human modification (deltas captured at the gates) as improvement signal.
- Token economics: own cost per story: per-invocation token and cost telemetry, model-mix tuning (fast tier for enrichment and test generation, top tier for code), and budget-aware prompt design.
- AI-native practice: work daily with agentic coding tools; the friction you hit is product input.

Education & Experience





- 5+ years of software engineering experience, including 2.5+ years building LLM-powered applications in real production use, not prototypes or side projects.
- Demonstrable experience with retrieval-augmented generation and context engineering in a production system.
- Experience building or operating systematic evaluation for LLM outputs (golden datasets, regression harnesses, rubric scoring).
- Experience in a startup or small-team environment is a strong plus.
- Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent demonstrable experience.

Technical Skills Required

- Strong Python and production experience with LLM provider APIs (Anthropic Claude family experience valued); structured outputs and JSON-schema validation.
- Prompt and context engineering with measurable iteration, exemplar selection, context budgeting, instruction design for consistency.
- Embeddings and vector search in production (pgvector or similar); code-aware chunking and indexing is a strong plus.
- Evaluation engineering: building harnesses, defining acceptance bars, wiring them into CI, and reading the trends.
- Token and cost optimisation: metering, model-tier routing, and cost-per-outcome thinking.
- Agent orchestration frameworks (LangGraph or similar) are a strong plus, as is SQL fluency for telemetry analysis.
- Practical, hands-on fluency with contemporary agentic coding tools (e.g., Claude Code or equivalent) is required.

📌 AI Engineer: Agents & Evaluation (Gurugram)
🏢 WOW Softech
📍 Gurugram

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: ai engineer: agents & evaluation (gurugram) / gurugram