At a glance
- Experience
- 1+ years experience
- Location
- Bangalore
About the role
- The Senior AI Evals Test Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK) or similar agentic frameworks.
- This role bridges AI development and reliable deployment, ensuring agents are safe, ethical, effective, and meet high-quality performance standards with a significant focus on Responsible AI and Safety Evals.
Responsibilities
- Design and build agent evaluation frameworks to evaluate LLM-based agents across correctness, latency, cost, safety, and task completion rate.
- Build synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions.
- Build golden datasets and test harnesses to curate domain-specific evaluation sets for Rubrik's product verticals.
- Establish eval-as-a-service by building and enhancing a centralized evaluation platform for self-service testing against standardized and custom benchmarks.
- Automate regression and drift detection by implementing CI/CD-integrated eval pipelines.
- Define eval taxonomies and rubrics to establish structured scoring criteria for agentic workflows.
- Instrument production observability by building real-time monitoring for agent quality in production.
- Benchmark and select models by running structured model comparisons across agent use cases.
- Analyze and interpret evaluation results, providing clear, actionable quality and safety reports to stakeholders and development teams.
- Provide technical leadership to engineers, including solutioning, design reviews, code reviews, pair programming, and mentoring.
Requirements
- Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience.
- 8+ years of overall software Test engineering experience, with 5+ years focused on test engineering, QA automation, or developer tooling.
- 2+ years working with ML/AI systems, building, testing, or evaluating models in production environments.
- 2+ years as a senior/lead, owning evaluation strategy, quality frameworks, and technical decisions for complex AI agent systems.
- 1+ years in designing or operating evaluation frameworks specifically for generative AI / LLM agents.
- Deep hands-on experience with LLM evaluation using frameworks and libraries such as LangSmith, DeepEval, LLM as a Judge, Promptfoo, RAGAS, and Hugging Face evaluation tooling.
- Strong understanding of agentic AI patterns including tool use, function calling, multi-step planning, and RAG.
- Proficiency in Python with strong scripting and tooling skills, including pytest and async patterns.
- Hands-on experience with CI/CD pipelines and Git-based workflows.
Preferred qualifications
- Experience evaluating agents in cybersecurity, data protection, or infrastructure domains
- Familiarity with Rubrik's product portfolio or similar enterprise data management platforms
- Hands-on with human-in-the-loop eval workflows
- Experience with model fine-tuning evaluation
- Background in building internal developer platforms or tooling teams
- Contributions to open-source eval frameworks or published work on AI evaluation methodology
- Drive red-teaming and adversarial testing for enterprise-grade agents
Skills
- Python
- TypeScript
- Java
- Selenium
- Playwright
- Anthropic Claude
- OpenAI
- Azure OpenAI
- DeepEval
- Promptfoo
- RAGAS
- LangSmith
Show all 15 skills
- LangGraph
- Kubernetes
- Jenkins
