flakebench AI

One good answer isn't the whole story

Repeat the same agent task and keep the failures, not just the best example.

Runs the same agent task many times and measures how often it fails.

Python

The problem

An agent is a model in a loop that can call tools before answering. A good demo only shows one run. flakebench repeats tasks, checks the answers, and records where runs fail. It can use a chat-completions endpoint or an offline mock.

How it works

Same taskRepeat runs
Model ↔ toolsDeadline + checker
ReportPasses + failures

Repeated runs reveal variation that a single answer cannot show.

Describe what counts as success
A YAML suite, a small text configuration file, gives each task a prompt, allowed tools, and a checker. A checker can compare exact text, match a pattern, validate JSON structure, or call a Python function.
Run the loop with a deadline
Each run gets its own process. The model can call Python tools in order, then receive their results and continue. The deadline includes startup, model calls, tools, and checking. A step limit caps how many model calls it can make.
Keep the whole set of outcomes
The runner writes one JSON record per run and a Markdown report. Failures distinguish wrong answers, malformed tool arguments, tool errors, timeouts, step limits, and model errors. Flakiness means a task passed on some runs and failed on others.

Decisions & tradeoffs

Show uncertainty

Pass rates include Wilson intervals, a range describing uncertainty in a proportion. Those intervals do not prove the tasks are independent or representative of every real use.

A deadline is not a security sandbox

Python tools run with the user's permissions, so suites and plugins need to be trusted. The offline mock injects failures deliberately; its report measures a simulation, not model quality.

Try it yourself

From a fresh checkout on Linux or macOS, use Python 3.10+ and a virtual environment. This small run uses only the included mock and trusted example tools.

python3 -m venv .venv
. .venv/bin/activate
pip install -e '.[dev]'
flakebench run examples/heldout.yaml --model mock --seed 42 --runs 2

Open report.md in the new results directory. All reported outcomes in this run are simulated. Two runs are useful for a quick look at the files, not a strong reliability estimate.

Sources & scope

Based on the local README, run loop, and report calculations. No simulated score is presented as a real-model result.