Job Summary

Are you passionate about leveraging technology to automate and transform business processes? Do you have a deep understanding of AI and test automation and a desire to drive innovation in enterprise systems? If so, the Tech Enablement and Automation team at TEMAIL_ADDRESS is looking for a Lead Consultant in Test Automation to help scale our automation capabilities and quality practices across the organization.

As a Lead Consultant in Test Automation, you will be responsible for driving the quality and reliability of automation solutions across the enterprise. In this role, you will work closely with engineers and cross-functional teams throughout the full application lifecycle, ensuring that systems are thoroughly tested, resilient, and aligned with business needs from development through to production. Are you innovative and curious about new technology that can make business processes easy and more productive?

Come join our team!

Responsibilities

- As a Lead Consultant you will play a pivotal role in shaping and advancing the company s enterprise-wide quality assurance and engineering strategy for AI powered systems and solutions. You will leverage both your technical expertise and leadership capabilities to oversee the delivery of scalable, high-quality AI and automation solutions.

Your responsibilities will include driving resolution of complex testing challenges, leading the implementation of critical enhancements, and spearheading modernization initiatives to ensure our AI and automation frameworks and practices are future-ready and aligned with business objectives.

- Lead with influence by promoting a culture of engineering excellence, responsible AI, and continuous improvement.

- Develop and maintain automation frameworks tailored for testing AI/ML workflows, including model training, inference, and integration.

- Create and execute automated regression, performance,



and data validation tests to ensure the stability and quality of AI-driven systems.

- Implement AI-specific test approaches such as synthetic data generation, result explainability validation, and continuous model monitoring.

- Drive innovation by identifying opportunities to embed automation and intelligence into the testing lifecycle using GenAI tools and frameworks.

- Partner cross-functionally to define and track AI quality metrics (accuracy, precision, recall, fairness) and integrate them into CI/CD pipelines.

- Lead and mentor global and cross-functional QA teams, promoting best practices and continuous improvement.

- Collaborate with senior leadership, product owners, architects, and development teams to ensure quality is built into all stages of the development lifecycle.

- Drive the implementation of test automation strategies and continuous testing practices.

- Identify test process improvement opportunities and lead initiatives to enhance QAE maturity and efficiency.

- Oversee risk identification, mitigation planning, and resolution related to QA deliverables.

- Excellent communication and stakeholder management skills.

- Exposure to Power BI, Splunk, or other monitoring and reporting tools for real-time visibility into test execution and system behaviour.

- Champion an automation-first mindset in test execution and continuous delivery strategies.

Requirements

- Deep understanding of LLMs, Generative AI, RAG, and Agentic AI architectures.

- Hands-on experience in AI evaluation frameworks (RAGAS, DeepEval, LLM-as-a-Judge, LangFuse).





- Expertise in Prompt Validation, Prompt Injection Testing and Adversarial Testing (Red Teaming).

- Knowledge of Data Validation, Data Quality, Data Drift, and Training/Inference Pipeline Testing.

- Deep hands-on experience with a broad range of test automation tools and technologies, including UI and functional testing with Playwright, Cypress; API testing with Postman; and performance testing with k6, JMeter.

- Strong understanding of supervised/unsupervised learning.

- Experience with ML platforms and tools and cloud AI services.

- Knowledge of DevOps practices and CI/CD tools like Jenkins, GitHub Actions, or Azure DevOps.

- Excellent analytical, debugging, and communication skills with a passion for quality and innovation.

Basic Qualifications

- Bachelor s or Master s degree in Computer Science, Data Science, Information Technology, or a related field.

- 7+ years of test automation experience, including at least 2 years focused on AI/ML testing.

- Proven track record in designing and executing automated test suites for complex, distributed, or AI-based systems.

- Strong programming and problem-solving skills with experience in Python, JS/TS, or similar languages.

Preferred Skills

- Experience with AI quality tools such as DeepChecks, Great Expectations, DataFold.

- Understanding of MLOps and automated model lifecycle management.

- Certification in AI/ML, Test Automation, or Cloud Technologies (e.g., AWS Certified Machine Learning, ISTQB Advanced Test Automation Engineer).

Job Details

- Will this position support sponsorship: No

- Travel percentage for this position: 0-10%

Disclaimer: This job posting has been aggregated from external source. Role details, content, and availability are subject to change. Applicants are advised to confirm the latest information directly on the company website before applying.

📌 Test Automation Lead - AI Testing (Hyderabad)
🏢 Eli Lilly
📍 Hyderabad