All updates

One run, no answer key: finding where frontier models fail

Can you catch a wrong answer from one run, with no answer key—and use it to generate tasks that expose new failures?

Three overlapping blue bell curves on white: one broad, low curve and two narrower, taller curves with slightly offset peaks.
Spread and agreement. Conceptual artwork inspired by probability distributions; not measured results or an assumed model of agent behavior.

Can you tell when a frontier model got a research task wrong, with no answer key, from a single run? Can you then subsequently generate tasks blindly while still exposing failure modes? We spent a few weeks on that question playing around with JEV: the short answer looks like yes, and one run turns out to do something two runs can obscure!

At first, we started with older models such as GPT-5.5, GPT-5.6 Luna, and Claude Opus 4.6, but were surprised to find the proof of concept worked on Claude Opus 5.5. Note, however, we ran all of these at low effort (we aren’t made of tokens, okay!).

Reading the numbers. Most scores below are AUROC. Pick one failing run and one correct run at random: AUROC is the chance the detector rates the failing one as riskier. 0.5 is a coin flip; 1.0 is perfect. For pair-level results, the question is whether the pair contains a wrong run.

  • Asking twice works, with a catch. When two runs disagree, at least one is usually wrong. But it takes a second run, and once the runs agree the disagreement judge cannot separate correct pairs from failing ones.
  • One run is enough to start. JEV reads a single finished run and estimates whether the answer is right. No key, no second run, about $0.00015 a read. So it works on traces you’ve already logged.
  • It finds the failures disagreement can’t. When both runs agree, one JEV read still ranks the wrong ones higher: 0.64 to 0.72 across all four models, against 0.50 for the judge.
  • Fancy didn’t transfer. Classifiers that won cross-validation on training benchmarks fell to near chance on new tasks. Plain JEV held at 0.69 on the generated tasks.
  • The pieces close a loop. Generate tasks, run once, score with JEV, confirm, climb. We tested the parts on simple generated tasks with a planted answer.

The problem: you can’t grade what you can’t check

The failures worth finding live where nobody has an answer key yet: new research questions, long agentic analyses, tasks no model has seen. So we asked a narrower question. Without a key, can you tell which runs of a frontier model are wrong, cheaply enough to find them at scale?

The bet was that wrong answers leave fingerprints. Run the model twice and the runs may split. Read its work and it may hedge. Watch its effort and it may stall or grind. Each of these can be measured without knowing the answer.

One or two runs to detect, more to test. Every detector here needs one run (JEV) or two (the pair-judge). To get more examples to score, we ran each task several times and graded every run against its answer key. On the main real-benchmark set, GPT-5.5, GPT-5.6 Luna and Claude Opus 4.6 ran each task four times. Claude Opus 5.5 ran its real tasks twice and its generated tasks six times. One-run scores treat each run as an example. Two-run scores treat each pair of runs of the same task as one: four runs give six pairs, two runs give one, and the six generated runs were split into three separate pairs.

1. Ask twice: a strong alarm with a blind spot

The simplest signal won early. Run the model twice on the same task, then ask an LLM judge one question: do these two conclusions agree? No answer key, no rubric, about half a cent per pair.

VARIANCE ANALYSIS / FIGURE 01

A strong alarm. An incomplete one.

How often does a pair contain at least one wrong run? Select what the judge observed.

Pairs containing a wrong run (%)

GPT-5.5

147 real benchmark tasks
Failure rate
93%

Claude Opus 4.6

147 real benchmark tasks
Failure rate
94%

Claude Opus 5.5

312 real benchmark tasks
Failure rate
73%

Claude Opus 5.5

32 generated tasks
Failure rate
100%
Conditional failure rates, not AUROC. Denominators are pairs with the selected judge call. Task counts describe the source pools; not all tasks produced scored pairs. Luna is not reported in this comparison. Source data ↗

When the runs split, something is usually wrong, and on tasks with exactly one right answer, almost always. As an alarm it’s hard to beat. But it has two built-in limits:

  • It needs a second run. That’s a whole extra agent rollout: minutes of compute, tool calls and tokens, which costs far more than the judge. A trace you logged last month may not be reproducible under the same conditions today.
  • It’s blind when the runs agree. The judge gives every agreeing pair the same score, so among them its ranking is a coin flip by construction. On Opus 5.5’s real benchmarks, 57 of 98 failing pairs looked like agreement, and in 47 of those both runs were wrong. Same mistake, twice.

2. One run, read by JEV

So we asked whether one run could be enough. With one run there’s nothing to compare, so the signal has to come from the run itself.

The plainest version won: a single JEV read. JEV reads the task, the run’s visible work and its final answer, with no key, and returns a probability that the answer is right. We used one minus that probability as the risk score. No features, no fitting.

  • Pair-judge (we used GPT-6 Sol): compares the conclusions of two runs. About $0.0054 and 2.7 seconds per pair, on top of the second rollout it needs.
  • JEV: reads one finished run. About $0.00015 and a quarter of a second, roughly 36 times cheaper than the judge alone, with no second rollout.

JEV asks every run the same fixed question about whether it finished correctly and returns probabilities over the answers, with no rubric or answer key. The read is also stable: when we asked JEV twice about the same 24 run snapshots, its probabilities barely moved. Stability does not establish calibration, though: use the score to rank runs for review, not as a pass/fail cutoff.

One run is a different regime, not just a cheaper one. Nothing has to be rerun, so you can score traces you already have: old agent logs, an eval you ran once, last month’s analyses.

To try it on your own logs, we’ve released Trace Variance Analyzer. It reads Codex and Claude Code logs, runs one JEV read per run, can run the pair-judge on runs of the same task, and scores both against any labels you have. The code is linked at the top of this post.

3. Breaking the tie

This is the result we care about most. Here are the two detectors side by side on real benchmark tasks. Switch between all pairs and only the pairs where both runs agree.

VARIANCE ANALYSIS / FIGURE 02

Agreement can hide a wrong answer.

One JEV read still ranks failures when the two-run judge has no disagreement to detect.

AUROC · higher is better

GPT-5.5

142 tasks
JEV · one run
0.65
Judge · two runs
0.50

GPT-5.6 Luna

141 tasks
JEV · one run
0.72
Judge · two runs
0.50

Claude Opus 4.6

140 tasks
JEV · one run
0.68
Judge · two runs
0.50

Claude Opus 5.5

312 tasks
JEV · one run
0.64
Judge · two runs
0.50
57 / 98

Failing pairs looked like agreement on Opus 5.5’s real tasks. In 47 of those pairs, both runs were wrong.

How to read AUROC

AUROC measures ranking, not accuracy. A score of 0.65 means a randomly chosen failing example is ranked riskier than a correct example 65% of the time. These scores do not establish calibrated failure probabilities.

Pair-level AUROC; JEV reads run A only. Scored within benchmarks, with whole benchmarks held out. All solvers at low effort; task pools differ. Point estimates only—confidence intervals are not supplied in the source JSON. Source data ↗

On all pairs, the judge is the stronger alarm for the two GPT models and about level with JEV on Opus 4.6. Once the runs agree, it has nothing left to say. One JEV read of one run keeps ranking failures above chance for every model. On Claude Opus 5.5, the newest model here, the single read edged out the two-run judge even across all pairs: more than half of its failing pairs were quiet.

So one read does two jobs at once. It needs only one run, and it still separates failures among the pairs that agree. That’s the part we find exciting. Discovering that a model makes the same mistake twice takes several runs: you only know a wrong answer is quiet because a second run agreed with it. Flagging it takes one.

4. More signals, more overfitting

Before landing on the plain read, we tried a lot of clever things: run time, token counts, tool use, answer length, entropy over the model’s likely next steps, step-by-step reads of the trace, follow-up questions, etc. We fed them to logistic regressions, random forests and gradient boosting. Cross-validated on real benchmarks, with whole benchmarks held out, they looked like progress.

To see how far that progress traveled, we took each model’s 10 best classifiers by cross-validation, out of about 220, and scored them further and further from where they were picked: on a real benchmark left out of the search, on a benchmark the search never saw (DiscoveryBench), and on new tasks we generated.

The lead lasted only as long as the tasks looked like the training data. On a left-out benchmark from the same pool, the winners matched plain JEV (0.70 against 0.71). On DiscoveryBench they fell to 0.56, and on the generated tasks to 0.52. For Opus 5.5 they didn’t just drop to chance there; they pointed the wrong way (0.23). JEV dipped on DiscoveryBench too (0.59), but it was back to 0.69 on the generated tasks.

The figure below looks at individual signals and classifiers for Opus 5.5, rather than the top-ten averages above.

VARIANCE ANALYSIS / FIGURE 03

The task changes. The signal can flip.

On Claude Opus 5.5, signals that worked on real benchmarks often reversed on new generated tasks.

AUROC · below 0.50 means the ranking points the wrong way

Output tokens

AUROC
0.18

Run time

AUROC
0.26

Best trained classifier (cross-validation pick)

AUROC
0.34

32-feature trained classifier

AUROC
0.30

JEV, one read, no answer key

AUROC
0.69

Failing runs often ran longer on real benchmarks, but stopped early on generated tasks. Execution effort did not transfer as a reliable failure signal.

Single-run AUROC for Opus 5.5 only. Real results are scored within benchmark; trained rows are out of fold. Generated results are scored within generator family. These are individual published detectors, not the post’s average of the top ten classifiers. Source data ↗

The reason is mundane. On real benchmarks, failing runs grind: they run longer and write more. On our generated tasks, failing runs quit early. A detector that learns ‘long run means trouble’ has learned the benchmark, not the failure.

The same lesson showed up when we generated tasks. New tasks written to land in the same region of feature space as failure-heavy training tasks matched them on the surface but didn’t reliably fail. Looking like a failure isn’t the same as being one.

5. A test bed with a known answer

To test the loop, we needed generated tasks with confirmed answers by design on which the model fails, so we built a generator (also in the repo). Each task hands the model 16,384 rows of data on eight measured variables and asks for a treatment effect we planted.

We hide the confounding in a three-way interaction, where the product of three variables (smooth) or of their signs (thresholded) drives both who gets treated and the outcome, and let the other five variables act as distractors. Each hard task has a matched control with the same outcome but random treatment, so a failure points at the confounding rather than the data.

Before any model saw a task, two coded estimators had to recover the planted effect, a naive adjustment had to miss it by more than 0.4, and two other models (GPT-6 Sol and Astra, at high effort) had to solve it blind. Over six runs each, 28 of 32 hard tasks failed at least once, and the controls never did (0 of 96 runs).

VARIANCE ANALYSIS / FIGURE 04

Two task families. Two ways to fail.

Synthetic causal-inference tasks isolate a hidden three-variable interaction, with a planted answer and matched randomized controls.

Quiet failures

35 / 37

35 of 37 failing pairs agreed. Repeating the run usually reproduced a conclusion the judge considered consistent.

75 / 192Hard runs wrong
0 / 96Control runs wrong
0.68JEV AUROC on agreeing pairs, within family
Claude Opus 5.5, low effort. Each tile is one failing pair in the selected family. Selected counts come from Trey’s article; run totals are in the repository. These are different metrics: agreeing pairs for smooth tasks, flagged pairs for thresholded tasks. Source data ↗

The two flavours fail in different ways, which makes them a useful test of the blind spot. Thresholded tasks fail loudly: the runs split, and the judge flags 17 of 19 failing pairs. Smooth tasks fail quietly: 35 of 37 failing pairs agree. Scored within each flavour, one JEV read ranked the agreeing failures at 0.68.

Where this goes

The pieces fit into a loop where an answer key, or extra runs, are needed only for tasks that were already flagged.

VARIANCE ANALYSIS / FIGURE 05

Let each failure inform the next test.

Explore the proposed loop. Use inexpensive screening to focus the more expensive investigation.

Aim the next batch at what failed
Measured

Score with JEV

Read the completed trace and rank likely failures using 1 − P(correct). Use the ranking to choose what to review, not as a calibrated pass/fail threshold.

$0.00015Per JEV read in the study
~0.25 sPer completed trace
36×Cheaper than the pair judge alone
Conceptual workflow based on Trey’s diagram. Components were tested; sustained adaptive generation and specialist detector training remain future work. Screening costs exclude model rollouts and confirmation runs. Source data ↗

It’s cheap. A JEV read costs a fraction of a cent, so the expensive step in initial screening is the model run itself, and that happens once per task. Confirming flagged tasks can require additional runs or a check against the key.

It should get sharper as it narrows. Our detectors overfit when they crossed from one kind of task to another. A detector trained inside one task family never makes that jump. Scored within each generated family, the judge plus JEV already reached 0.88. A specialist trained there could screen new tasks in that family quickly and cheaply. That is the next test, not a result yet.

In use, the planted answers drop away. The no-key signal (JEV on one run, plus disagreement when there are two) picks which generated tasks to keep, and the generator is steered toward them, so each new batch needs only fresh runs to be scored.

The one thing we wouldn’t do is train the model itself on that signal. A model rewarded for scoring well on it could learn to fool it, for example by giving the same wrong answer every time, which is exactly the agreeing failure this post is about.

Caveats

  • Small held-out sets. The generated test has 32 hard tasks, and intervals on the agreeing-pair results start just above chance.
  • One kind of generated task. Both flavours are causal-inference problems. Every generated-task number is scored within one flavour, because pooling them inflated scores.
  • Different task pools. The three earlier models share one set of 147 real tasks (140 to 142 scored per model); Opus 5.5 used a larger pool of 312. Compare models with care.
  • JEV isn’t universal. It catches failures that leave a trace: contradictions, guesses, broken constraints. It misses traps whose evidence never gets printed, and on some model and benchmark combinations it pointed the wrong way.
  • Labels come from answer keys, not human review. Real-benchmark runs were graded against each benchmark’s key; generated tasks against the planted answer.
  • Low effort only. Disagreement is also less precise when a task has more than one defensible answer.

The idea that disagreement and self-assessment reveal errors is not new. What’s new here is the setting: long, tool-using research tasks, one cheap read of a single finished run, and failures that two runs agree on.