n

What You’ll DoAgentic AI Feature & Workflow Development

- Build and integrate AI-driven features using LLM APIs (OpenAI / Azure OpenAI, Anthropic, Gemini on Vertex AI)
- Design and implement tool-using agents (structured function calling, schema validation, retries, fallbacks)
- Build multi-agent workflows when appropriate (e.g., planner/worker, reviewer/critic, specialist routing) and know when a simpler architecture is better
- Create agentic workflows such as document understanding, extraction, reasoning over evidence, task automation, and multi-step decision support

- Own context engineering end-to-end:
- dynamic context assembly (retrieval + state + tool outputs)
- context budgeting and compression/summarization
- grounding strategies to reduce hallucinations and improve consistency

- Implement retrieval-augmented generation (RAG) and search workflows using off-the-shelf vector stores and embedding services
Evaluation, Quality & Iteration (Core)

- Establish evaluation frameworks for accuracy, reliability, and output quality
- Build task-specific eval suites: golden datasets, adversarial cases, regression tests, and rubric-based scoring
- Set up automated evaluation pipelines and release gates (CI/CD-friendly) tied to prompt/model/version changes
- Define and monitor online metrics (e.g., task success rate, human override rate, safety flags, latency, cost) and run experiments/A-B tests where appropriate
- Use LLM-as-judge responsibly: calibrate, validate, and pair with human labels when needed

Engineering, Integration & Observability

- Develop scalable backend services and APIs that incorporate AI functionality
- Integrate AI pipelines into existing cloud, microservices, and event-driven architectures
- Implement observability and analytics for all AI features (tracing, evaluations, prompt versioning, cost tracking) Example tooling:



Langfuse (and/or OpenTelemetry-compatible stacks)
- Ensure reliability, uptime, performance, and security of AI services
- Build internal tooling for evaluation, testing, prompt/version management, and safe deployment

Product & Collaboration

- Partner with product managers, designers, the Chief Architect, and domain SMEs to shape AI-first solutions
- Rapidly prototype concepts and iterate based on user feedback and measurable eval results
- Translate business problems into well-structured AI workflows without requiring ML model training
- Document system behavior, known failure modes, and operational playbooks

Governance & Safety

- Implement guardrails, checks, and fallback logic for safe and predictable AI behavior
- Help define and follow compliance, privacy, and responsible AI guidelines
- Design for safe tool execution (bounded actions, permissions, escalation paths, human-in the-loop review

What You BringCore Strengths (Required)

- Strong software engineering background (Python preferred) and experience shipping backend services
- Deep hands-on experience building agentic LLM systems from first principles: agent loops, tool interfaces, planning/replanning, memory/state, and failure handling
- Strong context engineering ability: retrieval strategies, routing, grounding, context budgeting, and long-context tradeoffs
- Strong evaluation discipline: golden datasets, regression gating, automated eval pipelines, and online monitoring




- Practical experience with LLM APIs (OpenAI/Azure OpenAI/Anthropic/Gemini) and AI orchestration frameworks

- Excellent debugging, systems thinking, and problem decomposition skills
- Comfortable operating in fast-paced, ambiguous environments with high ownership

Signals We Value

- You’ve shipped an LLM/agent system in production and can clearly explain:
- the failure modes you discovered
- the evals you built to catch regressions
- how you improved cost/latency while increasing quality
- how you monitored and iterated safely over time

- You keep up with industry developments (model releases, frameworks, best practices) and can translate them into pragmatic improvement

Nice to Have

- Experience with cloud platforms (AWS and/or GCP), microservices, and event-driven systems
- Experience with observability stacks (OpenTelemetry, Datadog, Honeycomb) and AI-specific tooling (e.g., Langfuse, Braintrust, HumanLoop, W&B; Weave)
- Experience with workflow orchestration for long-running jobs (Temporal, Celery, Airflow)
- Experience building enterprise AI features (permissions, auditability, compliance constraints)
- Experience with safety/policy layers (PII handling, prompt injection defenses, sandboxed tool execution)

Why Join Us

- Build core AI capabilities that directly impact users and product strategy
- Work on cutting-edge, real-world agentic systems—focused on applied engineering (no model training required)
- High ownership, fast iteration cycles, and robust cross-functional collaboration
- Competitive compensation and opportunities for rapid advancement

What Your First 90 Days Could Look LikeShip one production agent workflow end-to-end with:

- tracing + observability
- an offline eval suite with regression gates
- cost/latency targets and monitoring
- documented failure modes and fallback path

n

📌 Senior AI Engineer (Pune)
🏢 Onit
📍 Pune