Production experience with at least one major LLM API Anthropic Claude
OpenAI Azure OpenAI or Amazon Bedrock including the full surface chat completions streaming token counting context management retrybackoff and ratelimit handling
Deep familiarity with toolcalling functioncalling defining tool schemas parsing toolcall results multiturn tool loops and handling partial or malformed tool outputs gracefully
Experience with structured output constraining LLM output to a JSON schema validating it and building retry loops that reprompt on validation failure without thrashing
Understands context window management token budgeting across a multiturn conversation what to summarise vs what to keep verbatim how context window size affects latency and cost
Comfortable with multistep agentic loops planning toolcall observe replan cycles and knows how to detect and recover from loops hallucinated tool names and stuck states
Agentic Architecture Patterns
Designs humanintheloop workflows as firstclass system components not as an afterthought interruption but as a structured review step with a explicit proposal format diff and approvalrejection path that feeds back into the agents context
Understands the difference between orchestrator agents and subagents knows when to delegate to a specialised subagent vs handle in one prompt and designs the handoff contract between them
Designs stateful multisession agentic pipelines checkpoint and resume idempotent reexecution of steps audit trail of every agent action and human decision
Familiar with ReAct PlanandExecute and reflection patterns knows when each applies and what their failure modes are
Experience building providerabstraction layers wrapping LLM calls behind an interface so the pipeline is not coupled to one vendor knows how to handle providerspecific quirks token limits toolcall formats refusals at the abstraction layer
RetrievalAugmented Generation RAG and Ingestion
Builds production RAG pipelines chunking strategy by section by semantic boundary by token count embedding model selection vector store indexing retrieval and reranking
Experience ingesting heterogeneous source types PDFs Word documents structured JSONXMLXLSX and unstructured legacy system exports each with its own extraction and preprocessing strategy
Understands retrieval quality evaluation recallk mean reciprocal rank how to build a golden dataset and measure retrieval against it
Experience with at least one vector store Pinecone Weaviate pgvector
Chroma Qdrant including index management metadata filtering and hybrid search semantic keyword
Evaluation and AI Governance
Builds LLM evaluation frameworks defining what correct means for a given agent task building test sets running evals as a CI gate before promoting a prompt or pipeline change
Designs AI assurance audit trails every AIassisted action is captured what was proposed what the human approved with what version of the prompt and model queryable and immutable
Understands AI failure modes hallucination sycophancy prompt injection instruction following degradation at long context and how to defend against each in a production system
Familiar with model evaluation dimensions factual accuracy instruction following format compliance output safety and knows which matter most for configurationauthoring tasks
Languages and Frameworks
Proficient in Python for AI pipeline work async patterns Pydantic for structured output validation httpxaiohttp for LLM API calls pytest for eval harnesses
Proficient in TypeScriptNodejs for AInative API clients and agent integrations in the platforms designtime surface
Familiar with at least one agentic framework LangChain LangGraph
LlamaIndex CrewAI AutoGen Haystack or Anthropics agent SDK and can reason about what the framework buys vs what it obscures
Comfortable writing productiongrade tested observable Python services not just Jupyter notebooks knows how to package deploy and monitor an agent pipeline
Observability for AI Systems
Instruments agent pipelines with LLMspecific observability traces that capture prompt completion token counts latency tool calls and model version per step
Familiar with tools like LangSmith Langfuse Arize or equivalent for LLM trace capture and prompt debugging
Builds cost dashboards token spend per pipeline per model per run type