Ex-Google Cloud GenAI architect building benchmarks that check what an agent did, not just what it said.
Session
Workshop
Date
Tuesday, December 1
Level
Intermediate
// AI Engineering Skill /
Evaluation-driven development
// 01 / Bio
Andi Partovi is co-founder and CTO of Veris AI, which builds simulation environments where enterprise AI agents are tested and trained before they reach production. He leads the team behind Veris’s public agent benchmarks, including VAmoS Bench for voice agents, accepted to the ReMuCAI workshop at NeurIPS 2026, and FinAgent Arena, which tests agents on end-to-end financial work inside a fully simulated bank.
Andi spent nearly three years at Google Cloud working on machine learning and generative AI. He started as an AI and ML customer engineer, led the AI/ML specialist team for Australia and New Zealand, and then moved to San Francisco as a generative AI specialist solution architect, helping enterprises and startups design and deploy LLM applications. Watching those teams test agents by hand is what led him to start Veris.
Before Google, he built deep learning systems in healthcare and marketing. As co-founder and CTO of Cardiopin, he worked on detecting changes in heart failure from voice biomarkers, and as a senior research fellow at Monash University he applied deep learning to electronic health records to forecast patient outcomes. He also co-founded KeyLead Health and Metronome, a marketing mix modeling platform built on Bayesian and neural network models. He holds a PhD in computer science from the University of Melbourne.
// 02 / The session
Breaking agents on purpose: the art of better benchmarks
Abstract
Public leaderboards say little about how an agent will behave in your product. Agents work across many steps, call real tools, and deal with users who interrupt and change their minds, so grading a model’s reply misses most of what goes wrong. In Veris’s VAmoS Pro Bench, the best voice agent completed only 44.7% of 4,200 simulated customer-service calls, and agents often reported finishing actions whose tool calls had failed.
This hands-on workshop shows how to build a benchmark around your own agent. You’ll set up a simulated environment with stateful versions of the tools and services your agent depends on, create simulated users and scenarios that push it into edge cases, and score each run on what changed in the environment rather than on the transcript. It’s the same approach Veris uses for VAmoS Bench and FinAgent Arena, its simulated-bank benchmark for financial services agents.
The session moves in four parts: where public benchmarks fall short, building the environment and scenarios, measuring task outcomes, and making evaluations repeatable so you can compare models, prompts and components as your agent changes.
Co-led with Mehdi Jamei, co-founder and CEO of Veris AI. Mehdi was previously Director of AI at System and executive director of the nonprofit Bayes Impact, and holds a PhD in EECS and an M.S. in physics from UC Berkeley.
What you’ll learn
Where existing benchmarks fall short for real applications
How to build realistic test environments and challenging scenarios
Scoring each run on what changed in the environment
Making evaluations repeatable as your agent changes
AI DEV · BY DEEPLEARNING.AI
Join us at AI Dev
Meet the engineers shipping AI to production, and leave ready to do it yourself.