Associate Lead - Testing (QA + MLOps)

Experience: 10+ years

Location: Mumbai/Bangalore (Hybrid)

Skills: AWS, Amazon SageMaker, Amazon Bedrock, AWS Lambda, Amazon API Gateway, Amazon Kinesis, AWS Glue, Amazon S3, Amazon CloudWatch, Python, Scikit-learn, TensorFlow, PyTorch, LangChain, AWS IAM, AWS CI/CD, CodePipeline, CodeBuild, Terraform, CloudFormation

Key Responsibilities:
AI/ML & GenAI Testing Strategy (AWS Ecosystem)
Define testing approaches for AI systems built on AWS services such as:
• Amazon SageMaker
• Amazon Bedrock
• AWS Lambda
• Amazon API Gateway
• Amazon Kinesis
• AWS Glue
• Amazon S3
• Amazon CloudWatch
Design validation frameworks covering:
• Model accuracy & performance validation
• Data drift & concept drift detection
• Hallucination detection for LLMs
• Prompt robustness testing
• RAG validation (retrieval accuracy + grounding)
• Bias & fairness validation
• Safety & toxicity testing
MLOps Quality Engineering (AWS-Centric)
Validate the end-to-end ML lifecycle including:
• Data ingestion & feature pipelines
• Model training & hyperparameter tuning
• Model versioning & registry
• Deployment validation
• Canary & blue/green release validation
Work with AWS-native services such as:
• SageMaker Pipelines
• SageMaker Model Monitor
• SageMaker Feature Store
• Bedrock model evaluation workflows
• CloudWatch-based observability
Implement CI/CD quality gates for ML pipelines integrated with AWS DevOps tools.
GenAI & Agentic AI Testing




Define quality engineering approaches for:
• LLM-based applications using Amazon Bedrock
• Prompt engineering validation
• Multi-agent orchestration testing
• Chatbot & Voice bot conversational testing
• Intent classification validation
• Conversation drift & fallback validation
• API contract validation for LLM integrations
Build reusable evaluation harnesses for:
• BLEU / ROUGE scoring
• Embedding similarity scoring
• Response consistency
• Safety scoring frameworks
Framework & Capability Development
Design reusable AI testing accelerators
Create AWS-aligned AI test automation frameworks (Python-first)
Develop synthetic data generation strategies
Establish AI quality scorecards
Build an internal AI QA Center of Excellence
Client Engagement & Leadership
Lead AI/ML quality strategy workshops
Perform AI risk & readiness assessments
Present quality architecture to CXOs
Drive QA transformation programs
Mentor QA teams on AWS-based AI testing
Own delivery for AI testing engagements end-to-end
Must have skills:
Testing Expertise
• 8–12+ years in Quality Engineering
• Strong test strategy,



automation & governance experience
• Experience leading QA transformation initiatives
• Experience building frameworks from scratch
AI/ML & GenAI Expertise
• Deep understanding of ML lifecycle
• Experience testing ML models (NLP preferred)
• Hands-on experience validating LLM applications
• Strong understanding of:
• Prompt engineering
• RAG architecture
• Embeddings
• Bias & explainability
AWS AI/ML Expertise
• Hands-on experience with:
• Amazon SageMaker (training, deployment, monitoring)
• Amazon Bedrock (LLM integration & evaluation)
• S3-based data pipelines
• AWS IAM (security validation)
• CloudWatch monitoring
• Lambda & API Gateway integrations
• AWS CI/CD (CodePipeline / CodeBuild preferred)
• Understanding of:
• Infrastructure as Code (Terraform / CloudFormation)
• Observability in AI systems
• Cost monitoring for ML workloads
Technical Skills
• Python (mandatory)
• Experience with ML libraries (Scikit-learn, TensorFlow, PyTorch)
• Experience with LLM frameworks (LangChain, etc.)
• API & automation testing frameworks
• Git-based workflows
Leadership & Communication
• Strong client-facing communication
• Experience leading QA teams
• Ability to create strategy decks & solution proposals
• Solid stakeholder management
If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!

📌 Associate Lead Testing (QA + MLOps) (Mumbai)
🏢 Quantiphi Analytics Solutions
📍 Mumbai