15 Nov
|
Dataart Technologies
|
Bengaluru
15 Nov
Dataart Technologies
Bengaluru
Job highlights
- Expertise in Python with pytest, LangChain or similar AI agent frameworks, AI evaluation tools like Ragas and DeepEval, and experience with AI model validation and AI-assisted STLC workflows
- Design and build AI agents and workflows, develop automated AI evaluations and benchmarks, integrate LLMs and APIs into QE frameworks, test AI security and data pipelines, deploy AI/ML solutions on cloud platforms, and manage defect tracking and release readiness
Responsibilities
- Analyze requirements and define evaluation strategies, golden datasets, regression benchmarks, baselines, thresholds, and blocking metrics with domain-expert review.
- Design, build, and orchestrate AI agents, persona-based workflows, and reusable skills for the STLC using LangChain, LangGraph, or Semantic Kernel.
- Build automated evaluations and benchmarks for LLMs, agents, RAG, retrieval, tool trajectories, and contracts using Ragas, DeepEval, and promptfoo.
- Calibrate LLM-as-judge against human grading, manage non-determinism through repeated runs, and detect regression and drift.
- Write and maintain Python and pytest automation for AI, ML, API, data, and integration workflows.
- Use Cursor and agentic tools to analyze requirements, generate and review tests, and maintain evaluation suites.
- Validate deep-learning, reinforcement-learning, and predictive models using LangChain.
- Build AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, and defect triage, plus intelligent test selection and defect prediction.
- Integrate LLMs, APIs, embeddings, vector databases, retrieval pipelines, and enterprise data sources into QE frameworks using Python.
- Test REST and streaming APIs, agent tools, and RAG and MCP integrations, including authentication, errors, retries, and recovery.
- Test data and retrieval pipelines for schema integrity, lineage, idempotency, fallback, promotion gates, and rollback.
- Integrate evaluations, benchmarks, and agent workflows into QE frameworks and CI/CD on GitHub Actions, review Allure results, and keep TestRail aligned with automated coverage.
- Test prompt injection, jailbreaks, unsafe tool use, data exfiltration,
access control, PII controls, and guardrails.
- Measure accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnose distributed failures.
- Implement tracing, monitoring, and production evaluation using Langfuse, LangSmith, or a comparable AI observability platform.
- Deploy and operate QE agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.
- Deploy and validate AI and ML solutions on AWS or Azure, including secrets, database services, and cloud-native test infrastructure.
- Report, track, and verify defects in Jira, and work with domain experts, product, and engineering on traceable coverage and untested scope.
- Contribute to release readiness for the assigned product area without owning sign-off for the whole team.
Requirements
- Strong Python proficiency with pytest for test and evaluation automation.
- Hands-on experience building and orchestrating AI agents and agentic workflows with LangChain, LangGraph, Semantic Kernel, or comparable frameworks.
- Hands-on experience building AI evaluations and benchmarks with Ragas, DeepEval, promptfoo, or equivalent tools.
- Hands-on use of Cursor or agentic development tools for test generation and maintenance.
- Experience defining AI evaluation strategies, golden datasets, regression benchmarks, and LLM-as-judge calibration, including non-determinism and drift.
- Experience validating deep-learning, reinforcement-learning, and predictive models with LangChain.
- Experience building AI-assisted STLC workflows for requirements analysis, test design, script generation, execution, self-healing, failure analysis, defect triage, intelligent test selection, and defect prediction.
- Experience integrating LLMs, APIs, embeddings,
vector databases, and retrieval pipelines with automation frameworks in Python.
- Experience testing REST and streaming APIs, agent tools, RAG workflows, and MCP integrations.
- Experience testing data and retrieval pipelines for correctness, lineage, and secure promotion.
- Experience integrating AI evaluation and agent workflows with automation frameworks and CI/CD using GitHub Actions, Allure, TestRail, or equivalent tools.
- Experience testing AI security, privacy, access control, and guardrails.
- Experience measuring accuracy, relevance, groundedness, hallucination, retrieval quality, agent reliability, latency, throughput, token use, and cost, and diagnosing distributed failures.
- Experience with AI tracing, monitoring, and production evaluation using Langfuse, LangSmith, or comparable tools.
- Familiarity with AWS, Azure, or GCP cloud AI services and agent platforms for deploying and validating AI and ML solutions.
- Experience deploying and operating agents on Amazon Bedrock AgentCore or a comparable agent platform, with governance and human oversight.
- Experience managing defects through resolution in Jira or an equivalent tool.
- Ability to work with domain experts, product, and engineering on ground truth and coverage gaps.
- Experience contributing to release readiness for an assigned product area.
Nice to have
- Playwright and TypeScript experience for UI and API automation.
- Real-device and cross-browser execution using LambdaTest, Sauce Labs, or an equivalent platform.
- Knowledge of AI-driven test optimization and coverage analytics using SeaLights or an equivalent platform.
- Experience contributing reusable components to shared automation frameworks and reviewing automation code through Git and GitHub workflows.
- Experience managing test data and troubleshooting test environments needed for reliable execution.
- Experience triaging security findings and validating operational readiness, monitoring, rollback, and recovery controls.
- Experience producing audit-ready evaluation reports and improving shared evaluation patterns, automation libraries, and datasets.
Apply for the job
📌 Senior AI Quality Engineer (Bengaluru)
🏢 Dataart Technologies
📍 Bengaluru