Senior AI Quality Engineer (Bengaluru)

Senior AI Quality Engineer (Bengaluru)

15 Nov
|
Dataart Technologies
|
Bengaluru

15 Nov

Dataart Technologies

Bengaluru

Job highlights

- Expertise in Python with pytest, LangChain or similar AI agent frameworks, AI evaluation tools like Ragas and DeepEval, and experience with AI model validation and AI-assisted STLC workflows

- Design and build AI agents and workflows, develop automated AI evaluations and benchmarks, integrate LLMs and APIs into QE frameworks, test AI security and data pipelines, deploy AI/ML solutions on cloud platforms, and manage defect tracking and release readiness

Responsibilities

- Analyze requirements and define evaluation strategies, golden datasets, regression benchmarks, baselines, thresholds, and blocking metrics with domain-expert review.

- Design, build, and orchestrate AI agents, persona-based workflows, and reusable skills for the STLC using LangChain, LangGraph, or Semantic Kernel.

- Build automated evaluations and benchmarks for LLMs, agents, RAG, retrieval, tool trajectories, and contracts using Ragas, DeepEval, and promptfoo.

- Calibrate LLM-as-judge against human grading, manage non-determinism through repeated runs, and detect regression and drift.

- Write and maintain Python and pytest automation for AI, ML, API, data, and integration workflows.

- Use Cursor and agentic tools to analyze requirements, generate and review tests, and maintain evaluation suites.

- Validate deep-learning, reinforcement-learning, and predictive models using LangChain.

- Build AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, and defect triage, plus intelligent test selection and defect prediction.

- Integrate LLMs, APIs, embeddings, vector databases, retrieval pipelines, and enterprise data sources into QE frameworks using Python.

- Test REST and streaming APIs, agent tools, and RAG and MCP integrations, including authentication, errors, retries, and recovery.

- Test data and retrieval pipelines for schema integrity, lineage, idempotency, fallback, promotion gates, and rollback.

- Integrate evaluations, benchmarks, and agent workflows into QE frameworks and CI/CD on GitHub Actions, review Allure results, and keep TestRail aligned with automated coverage.

- Test prompt injection, jailbreaks, unsafe tool use, data exfiltration,



access control, PII controls, and guardrails.

- Measure accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnose distributed failures.

- Implement tracing, monitoring, and production evaluation using Langfuse, LangSmith, or a comparable AI observability platform.

- Deploy and operate QE agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.

- Deploy and validate AI and ML solutions on AWS or Azure, including secrets, database services, and cloud-native test infrastructure.

- Report, track, and verify defects in Jira, and work with domain experts, product, and engineering on traceable coverage and untested scope.

- Contribute to release readiness for the assigned product area without owning sign-off for the whole team.

Requirements

- Strong Python proficiency with pytest for test and evaluation automation.

- Hands-on experience building and orchestrating AI agents and agentic workflows with LangChain, LangGraph, Semantic Kernel, or comparable frameworks.

- Hands-on experience building AI evaluations and benchmarks with Ragas, DeepEval, promptfoo, or equivalent tools.

- Hands-on use of Cursor or agentic development tools for test generation and maintenance.

- Experience defining AI evaluation strategies, golden datasets, regression benchmarks, and LLM-as-judge calibration, including non-determinism and drift.

- Experience validating deep-learning, reinforcement-learning, and predictive models with LangChain.

- Experience building AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, defect triage, intelligent test selection, and defect prediction.

- Experience integrating LLMs, APIs, embeddings,



vector databases, and retrieval pipelines with automation frameworks in Python.

- Experience testing REST and streaming APIs, agent tools, RAG workflows, and MCP integrations.

- Experience testing data and retrieval pipelines for correctness, lineage, and secure promotion.

- Experience integrating AI evaluation and agent workflows with automation frameworks and CI/CD using GitHub Actions, Allure, TestRail, or equivalent tools.

- Experience testing AI security, privacy, access control, and guardrails.

- Experience measuring accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnosing distributed failures.

- Experience with AI tracing, monitoring, and production evaluation using Langfuse, LangSmith, or comparable tools.

- Familiarity with AWS, Azure, or GCP cloud AI services and agent platforms for deploying and validating AI and ML solutions.

- Experience deploying and operating agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.

- Experience managing defects through resolution in Jira or an equivalent tool.

- Ability to work with domain experts, product, and engineering on ground truth and coverage gaps.

- Experience contributing to release readiness for an assigned product area.

Nice to have

- Playwright and TypeScript experience for UI and API automation.

- Real-device and cross-browser execution using LambdaTest, Sauce Labs, or an equivalent platform.

- Knowledge of AI-driven test optimization and coverage analytics using SeaLights or an equivalent platform.

- Experience contributing reusable components to shared automation frameworks and reviewing automation code through Git and GitHub workflows.

- Experience managing test data and troubleshooting test environments needed for reliable execution.

- Experience triaging security findings and validating operational readiness, monitoring, rollback, and recovery controls.

- Experience producing audit-ready evaluation reports and improving shared evaluation patterns, automation libraries, and datasets.

Apply for the job

📌 Senior AI Quality Engineer (Bengaluru)
🏢 Dataart Technologies
📍 Bengaluru

Reply to this offer

Impress this employer describing Your skills and abilities, fill out the form below and leave Your personal touch in the presentation letter.

Subscribe to this job alert:

Get the latest job offers by email for: senior ai quality engineer (bengaluru) / bengaluru

Subscribe to this job alert:

Get the latest job offers by email for: senior ai quality engineer (bengaluru) / bengaluru