AI Evaluation Engineer (Education AI Systems)

Experience: 3 - 8 Years
Location: Bangalore / Hyderabad

We are looking for an AI Evaluation Engineer to build evaluation systems, tools, and workflows for AI-powered education products. This is a hands-on technical role focused on improving the quality, reliability, and performance of LLM-powered applications used for tutoring, assessments, content generation, teacher support, and student learning.

Key Responsibilities:

- Build and maintain evaluation pipelines for AI/LLM-powered applications.
- Develop Python scripts, notebooks, and automation tools for batch testing, model comparison, regression testing, and quality analysis.
- Design evaluation rubrics, golden datasets, test cases, annotation guidelines, and review workflows.
- Evaluate prompts, structured outputs, RAG pipelines, tool-calling workflows, and multi-step AI systems.
- Analyze AI outputs to identify hallucinations, reasoning errors, curriculum gaps, safety issues, and performance regressions.
- Collaborate with Engineering, Product, QA, Curriculum, and Subject Matter Experts to improve AI quality and release readiness.
- Track evaluation metrics and continuously improve AI performance through data-driven insights.

Required Skills:

- 3 - 8 years of experience in Software Development, AI Evaluation, Applied AI, QA Automation,



Data Analysis, EdTech, or related domains.
- Solid programming skills in Python.
- Experience working with LLMs, Generative AI, OpenAI APIs, Prompt Engineering, and AI evaluation workflows.
- Hands-on experience with JSON, APIs, structured outputs, datasets, and Jupyter Notebooks.
- Strong understanding of AI testing, debugging, regression analysis, and quality evaluation.
- Ability to create measurable evaluation rubrics, scoring frameworks, and test datasets.
- Excellent analytical, debugging, and problem-solving skills.
- Good understanding of curriculum alignment, educational content quality, learning objectives, grading, and instructional design.

Preferred Skills:

- Experience with RAG (Retrieval-Augmented Generation), vector databases, embeddings, and context management.
- Knowledge of LLM-as-a-Judge, human-in-the-loop evaluation, automated evaluation frameworks, and experiment tracking.
- Experience building annotation tools, dashboards, QA automation, or internal AI evaluation platforms.
- Familiarity with Git, SQL, notebooks, lightweight web applications, and data visualization tools.
- Experience working with AI-powered education products, tutoring platforms, assessments, or learning systems.

📌 Artificial Intelligence Engineer (Karnataka)
🏢 Hiringeye Solutions
📍 Karnataka