In this AVP-level role, you will be responsible for designing, implementing, and maintaining automated testing frameworks for AI systems. You will validate model accuracy, data pipelines, and software integrations to ensure high performance, reliability, and ethical compliance across the development lifecycle.
You will act as a crucial bridge between data science, software engineering, and quality assurance, working closely with data scientists, ML systems engineers, front-end engineers, and business leaders. If you are passionate about operationalizing complex models, creating highly reliable automated validation services, and defining the quality lifecycle for data science workflows at an enterprise scale, this role is a perfect fit.
Key Responsibilities & Competencies
Quality Framework Architecture: Design, build, and maintain scalable automated testing frameworks specifically tailored for AI/ML systems, including LLMs, RAG pipelines, and traditional ML models.
Model & Output Validation:



Design and execute automated evaluation pipelines to measure critical metrics such as hallucination rates, context retention, retrieval precision, semantic similarity, and answer relevance (using tools like RAGAs, DsPy).
Data Pipeline & ETL Auditing: Validate the quality, integrity, and schema compliance of data pipelines and ETL processes (PySpark, Hive, Kafka, Parquet, Iceberg) to ensure clean data flows into AI models.
CI/CD Integration: Integrate automated AI quality gates into enterprise CI/CD pipelines (using Tekton, Harness, UDeploy, Jenkins) to enable continuous testing and prevent regressions.
Risk Stewardship & Compliance: Lead adversarial testing (Red Teaming) to identify vulnerabilities like prompt injection, jailbreaking, and data leakage. Validate real-time safety guardrails to ensure compliance with ethical AI guidelines, corporate policies, and regulatory standards.
Operational Excellence & Observability: Implement monitoring and observability solutions (using MLF

📌 AI Quality Automation Engineer – Assistant Vice President (Pune)
🏢 Citi
📍 Pune