Job Summary The role will be responsible for independently evaluating AI/LLM systems, identifying potential risk vectors, designing robust evaluation frameworks, and establishing controls to support responsible and secure enterprise AI deployment.

Roles & Responsibilities

- Conduct RAG & GenAI risk assessments across end-to-end RAG pipelines, including knowledge-base ingestion, chunking, embeddings, vector retrieval, context grounding, and generation.
- Identify AI/LLM risks such as data leakage, prompt injection, context contamination, retrieval failures, and hallucinations.
- Design and execute LLM evaluation and benchmarking frameworks to assess hallucination rates, toxicity, robustness, model calibration, fairness, and bias.
- Curate golden/reference datasets and design evaluation test cases with reference answers.
- Implement and validate LLM-as-a-Judge evaluation pipelines.
- Establish AI governance and risk mitigation controls covering data preprocessing, pre-deployment validation gates, and post-deployment observability.
- Independently inspect, evaluate, and validate AI/LLM model outputs.
- Conduct technical audits of AI/LLM systems and assess model performance against defined evaluation criteria.
- Support the development of enterprise AI control frameworks covering data provenance, model sign-off, rate-limiting, and continuous drift monitoring.
- Evaluate automated AI assessment and hallucination detection tooling, including Ragas, TruLens, and DeepEval.

Qualifications

- B.Tech / M.Tech in Computer Science, Quantitative disciplines, or MCA.




- Robust quantitative and/or Computer Science foundation.
- Thorough understanding of AI/ML taxonomy and modern Generative AI workflows.
- Deep understanding of RAG architecture and associated risks, including vector search, embeddings, relevance scoring, and context grounding/grounding boundaries.
- Hands-on familiarity with reference-based testing, LLM-as-a-Judge frameworks, red-teaming fundamentals, and semantic similarity evaluation.

Experience & Skills 9 to 12 years of relevant experience

Required Technical Skills

- Strong expertise in Generative AI / GenAI and Large Language Models (LLMs).
- Strong understanding of Retrieval-Augmented Generation (RAG) pipelines.
- Experience in LLM evaluation, model validation, AI risk assessment, and AI governance.
- Understanding of AI risk vectors including prompt injection, hallucination, data leakage, and retrieval failures.
- Familiarity with enterprise LLM orchestration and deployment platforms such as Amazon Bedrock, Azure OpenAI Service, and Vertex AI.

Preferred / Good-to-Have Skills:
- Working knowledge of Python for querying APIs, parsing JSON outputs, and independently auditing model evaluation scripts.
- Practical experience with NLP evaluation metrics such as ROUGE, BLEU, BERTScore, and semantic distance metrics.
- Ability to construct enterprise AI control frameworks covering data provenance, model sign-off, rate-limiting, and continuous drift monitoring.
- Familiarity with automated hallucination detection and AI evaluation tools such as Ragas, TruLens, and DeepEval.

📌 AI Model Governance Specialist (Risk) (Navi Mumbai)
🏢 ICICI Securities
📍 Navi Mumbai