Position Overview

AriveGuru is seeking a Data Scientist with combined expertise in Artificial Intelligence and Quality Assurance. The selected candidate will be responsible for developing data science and machine learning solutions and for ensuring that every AI-driven capability we release meets defined standards of accuracy, reliability and consistency.

This is a hybrid position. Candidates must demonstrate working proficiency in both AI/data science and software quality assurance practices.

Key Responsibilities

AI and Data Science

- Develop, test and refine machine learning and LLM-based solutions, including retrieval-augmented generation (RAG), classification and information extraction
- Perform data analysis, cleaning and preparation to support model development and evaluation
- Design and iterate on prompts using structured, repeatable testing methods

AI Quality Assurance and Evaluation

- Design evaluation datasets, reference (“golden”) answers and test cases to measure the accuracy, relevance and consistency of AI outputs
- Define quality metrics, acceptance criteria and release thresholds for AI-driven features
- Build automated evaluation and regression-testing pipelines in Python, integrated into the development and release workflow
- Identify, document and report failure modes, including hallucinations, incomplete or incorrect responses, bias and formatting errors, and coordinate with engineering teams on resolution
- Apply LLM-as-a-judge methods and calibrate automated scoring against human review
- Monitor production performance, detect model drift and quality degradation, and escalate issues promptly
- Validate the completeness,



consistency and correctness of datasets used by our systems

Collaboration and Reporting

- Partner with backend, full stack and product teams to establish quality standards and release criteria
- Prepare clear dashboards, reports and documentation of evaluation results for technical and non-technical stakeholders

Required Qualifications

The position requires proficiency in both AI/data science and quality assurance. Applicants with experience in only one area will not be considered.

AI and Data Science

- Bachelor’s degree in Computer Science, Data Science, Statistics, Mathematics or a related discipline
- Approximately 2 years of hands-on experience in data science, machine learning or AI projects
- Strong proficiency in Python and its data libraries (Pandas, NumPy, Scikit-learn)
- Good working knowledge of SQL and relational databases such as PostgreSQL
- Sound understanding of machine learning fundamentals, statistics and evaluation metrics (precision, recall, F1 score, etc.)
- Practical experience with large language models, including prompt engineering, RAG, and an understanding of hallucination and grounding

Quality Assurance

- Working knowledge of QA fundamentals: test planning, test case design, edge-case analysis, defect reporting and regression testing




- Experience testing AI or ML outputs for accuracy, consistency and bias
- Ability to build automated tests and evaluation pipelines in Python
- Experience in data quality validation
- Ability to define pass/fail criteria and release standards
- Robust attention to detail and a disciplined, evidence-based approach to verifying results

Preferred Qualifications

- Experience with evaluation frameworks such as Ragas, DeepEval or LangSmith
- Familiarity with LangChain or LlamaIndex
- Knowledge of test automation and API testing tools (Pytest, Postman, Selenium or equivalent)
- Experience with Git, CI/CD pipelines and AWS
- Data visualization skills (Matplotlib, Plotly, Power BI or equivalent)
- Exposure to NLP, embeddings and vector databases

Desired Attributes

- Analytical, detail-oriented and methodical
- Strong written and verbal communication skills
- Ability to work independently and collaboratively in a fast-paced environment
- Commitment to continuous learning, including effective use of modern AI tools
- High sense of ownership and accountability for quality

What We Offer

- Opportunity to work on production-grade AI systems where quality standards directly influence outcomes
- Exposure to the complete lifecycle of AI development, evaluation and deployment
- Defined career growth toward senior data science and AI quality leadership roles
- Close collaboration with the leadership and engineering teams

Pay: ₹500,000.00 - ₹800,000.00 per year

Benefits:

- Paid sick time
- Paid time off
- Provident Fund

Work Location: In person

📌 Data Scientist – AI & Quality Assurance (Bengaluru)
🏢 AriveGuru Technology solutions
📍 Bengaluru