15 Nov
|
Brillio
|
Bengaluru
Consultant
Primary Skills
- CI/CD Pipeline, React (React Context / Redux), Django / Flask / FastAPI, Docker, CSS/SCSS/LESS, PyTest, PostgreSQL / MongoDB / Redis, Kubernetes, REST API / GraphQL
Job requirements
- Role: QA Engineer - GenAI Agents and GitLab Duo
- Design and execute functional, integration, regression, security, and performance tests for GenAI agents and multi-agent workflows.
- Validate application and infrastructure code interrogation results for accuracy, evidence, severity, false positives, and false negatives.
- Test GitLab Duo, AI Catalog agents, shared CI/CD templates, pipeline triggers, reports, and security gates.
- Develop automated Python-based test suites and evaluation datasets to assess agent reliability, consistency, security, and overall quality.
Key Responsibilities 1. AI Agent and Workflow Testing
- Create and execute test plans for GenAI agents and agentic workflows.
- Validate single-agent, multi-agent, and orchestrated workflows.
- Verify agent instructions, tool usage, task routing, handoffs, and termination conditions.
- Test agent behaviour for valid, invalid, incomplete, and ambiguous inputs.
- Validate error handling, retry mechanisms, timeouts, and fallback scenarios. 2.
Application Code Interrogation Testing
Validate the application code interrogation agent against:
- Security vulnerabilities and OWASP Top 10 risks.
- Insecure coding practices.
- Improper authentication and authorization.
- Sensitive-data exposure and secret leakage.
- Missing input validation and error handling.
- Resiliency issues such as missing timeouts, retries, bulkheads, and circuit breakers.
- Incorrect logging, monitoring, and exception-handling practices. Verify that each finding contains accurate:
- File name and line number.
- Issue description and technical evidence.
- Severity and confidence.
- Business impact.
- Recommended remediation.
- Classification as confirmed or potential finding. 3.
Infrastructure Code Interrogation Testing
Test the infrastructure interrogation agent against Terraform, Kubernetes, Helm, YAML, and CI/CD configuration files. Validate detection of:
- Single points of failure.
- Inadequate high availability and scaling.
- Insecure storage and missing encryption.
- Excessive network access.
- Unrestricted communication between security zones.
- Missing identity and access controls.
- Inadequate logging, monitoring, and alerting.
- Insecure secrets management.
- Misconfigured cloud and container resources.
- SDLC Agent Validation Support testing of AI agents used across the software development lifecycle, including:
- Requirement and story decomposition.
- UX/UI specification generation.
- OpenAPI contract generation and governance.
- Architecture validation.
- Backend and frontend implementation assistance.
- Dependency and open-source package curation.
- Unit, integration, contract, and end-to-end test generation.
- Security review.
- Infrastructure planning and implementation.
- Triage and Finding Validation Validate the interrogation-triage agent for:
- Finding normalization.
- Duplicate detection and deduplication.
- Correct categorization.
- Severity and priority assignment.
- Ranking based on exploitability, exposure, and blast radius.
- Correlation of findings from application and infrastructure agents.
- Accurate consolidation of related findings.
- Avoidance of false positives and unsupported conclusions.
- GitLab Dedicated CI/CD Integration Testing
- Validate agent execution through GitLab Dedicated pipelines.
- Test integration with shared CI/CD templates.
- Validate pipeline triggers for pushes, merge requests, and scheduled executions.
- Verify pipeline status, security gates, and failure behavior.
- Validate generated reports, artifacts, logs, and auditability.
- Test repository permissions and agent access controls.
- Verify behavior across different programming languages, repository structures, and infrastructure projects.
- Test Automation
- Develop automated test cases using Python and suitable testing frameworks.
- Build reusable test datasets containing secure, vulnerable, and intentionally incorrect code.
- Automate regression testing for prompts, tools, agents, and
📌 Lead Quality Engineer (Bengaluru)
🏢 Brillio
📍 Bengaluru