JailbreakBench does something most benchmarks do not: it publishes its artifacts. Not just the attack prompts, but the target model's actual responses, frozen in a repository at a known commit. The leaderboard ranks 18 attack and target-model pairs by attack success rate.
Producing that rate requires one more step. Something has to read each response and decide whether it constitutes a jailbreak. In JailbreakBench that something is Llama-3-70B, prompted a particular way, with its output read by a particular parser.
Nothing about an attack changes when you change the judge. The responses are files on disk. So if the ranking moves when only the judge moves, the ranking was never a property of the attacks.
It moves. Kendall's tau against the published ordering falls as low as 0.115. Four of the five alternate judges push it below the pre-registered threshold of 0.80. And the sharper of the two results is not the reordering at all: the judge-induced spread on a single entry reaches 0.82 ASR, against a smallest adjacent gap on the published leaderboard of 0.01. The measurement wobbles, under one declared dial, by 82 times the resolution the leaderboard reports at.
What was frozen, and when
The design was committed before any scoring was run, and the scoring script was committed separately before its first live run, so the gap between design and execution is visible in the repository history rather than asserted here.
Pre-registration v1.1 landed at c44d9211 on 2026-09-16 with no scoring performed. It went through five revisions before the analysis ran, every one of them logged in the document's own changelog. Two revisions are worth naming because they are the kind usually made silently: v1.2 corrected nine errors found by reading v1.1 against the pinned sources while writing the scoring script, and v1.4 declared a runtime deviation for one judge before that judge's pass rather than after it. The final document is v1.5, SHA-256 7f2bd31bf198554a9ceee7cc8300e1750e23d60ac98408ccd48384e23795b6e5.
Two thresholds were registered. H1 holds if Kendall's tau between the published ranking and at least one alternate configuration falls below 0.80. H2 holds if the judge-induced ASR spread on at least one entry exceeds the smallest gap between adjacent entries on the reference leaderboard. The pre-registration states explicitly that every combination of supported and unsupported is publishable, and that the study is not designed to produce a particular result. That sentence is load-bearing, because one of the four predictions failed and is reported below.
The setup
Artifacts from the JailbreakBench repository at commit 909e68c01d94222b8ad2e397a017e2e12e2adb73: 18 entries, 100 behaviors each, with target responses stored. Two dials.
Dial 1, the judge. The reference is Llama-3-70B as shipped. Alternates are Llama Guard 1 (from the stored labels), plus four local models run at q8_0 through Ollama at temperature 0 with num_ctx 8192: Llama-3-8B-Instruct, Qwen2.5-7B-Instruct, Mistral-7B-Instruct, and Gemma-2-9B-It. Four model families, chosen to span them rather than to find a winner.
Dial 2, the parser. Two levels. shipped is classifier.py at commit 23dbdf6b, the file audited in JailbreakBench issue #50. corrected is the same file with the eight defects in that audit repaired per its stated directions. The cited line numbers span three code paths, so the correction is three functions, not one. One defect was deliberately left unrepaired: a genuine refusal worded outside the parser's 13 listed phrases is a coverage limit of a fixed phrase list rather than a matching bug, and extending that list would have been an undeclared dial.
That audit explicitly scoped out re-scoring the published artifacts. This experiment runs it. Ten configurations, 10,800 scored rows.
H1: the ranking does not survive the judge
Kendall's tau between each configuration's ranking of the 18 entries and the reference ranking. The pre-registered threshold is 0.80.
| Configuration | Kendall tau | Entries | Above 0.80 |
|---|---|---|---|
| gemma-2-9b-it, shipped | 0.832 | 18 | yes |
| gemma-2-9b-it, corrected | 0.832 | 18 | yes |
| llama-guard-1, shipped | 0.774 | 16 | no |
| qwen2.5-7b-instruct, shipped | 0.769 | 18 | no |
| qwen2.5-7b-instruct, corrected | 0.769 | 18 | no |
| mistral-7b-instruct, corrected | 0.637 | 18 | no |
| mistral-7b-instruct, shipped | 0.308 | 18 | no |
| llama3-8b-instruct, shipped | 0.115 | 18 | no |
| llama3-8b-instruct, corrected | 0.115 | 18 | no |
H1 is supported. Minimum tau is 0.115. Only Gemma-2-9B clears the threshold, and it clears it by 0.032.
A tau of 0.115 is worth sitting with. It is not a slightly different ordering. It is close enough to zero that the ranking produced by Llama-3-8B carries almost no information about the ranking produced by Llama-3-70B, on the same frozen responses. On the 17-entry set excluding the one thinly populated PAIR entry, that figure drops further, to 0.023.
H2: the spread is larger than the leaderboard's own resolution
This is the result that should change how the number is read.
The smallest gap between adjacent entries on the reference leaderboard is 0.01 ASR. One percentage point separates two neighbors, and the leaderboard prints that separation as if it were a fact about the attacks.
The largest judge-induced spread on a single entry is 0.82, on JBC/manual/gpt-3.5-turbo-1106. Depending on which judge reads the same 100 stored responses, that entry's attack success rate ranges across 82 percentage points.
It is not one outlier. All 18 entries have a judge-induced spread exceeding the smallest adjacent gap. The JBC manual entries are the widest (0.82, 0.81, 0.78, 0.68), which makes sense: manually written jailbreaks produce the most ambiguous responses, and ambiguity is exactly where judges diverge. The narrowest is PAIR against Llama-2-7B at 0.04, which is still four times the smallest gap.
H2 is supported, at a ratio of 82 to 1. The honest reading is that a leaderboard reporting to two decimal places is reporting about two orders of magnitude more precision than the measurement supports under a single declared dial.
The parser dial moved exactly one judge, and moved it a long way
Across the four local judges, the shipped and corrected parsers disagreed on exactly one.
| Judge | Rows scored | Rows that flipped | Direction |
|---|---|---|---|
| llama3-8b-instruct | 1,637 | 0 | n/a |
| qwen2.5-7b-instruct | 1,637 | 0 | n/a |
| gemma-2-9b-it | 1,637 | 0 | n/a |
| mistral-7b-instruct | 1,637 | 534 | all one way |
Every one of the 534 flipped rows went the same direction: the shipped parser read the response as not jailbroken, the corrected parser read it as jailbroken. Zero went the other way. The cause is mundane, and that is the point. Mistral writes its verdict with a capitalization the shipped parser does not match.
A parser defect that is literally invisible on three judges silently suppresses a third of the positive calls on the fourth. And it is not a rounding difference: Mistral's tau against the reference moves from 0.308 to 0.637 on the correction alone. A capitalization bug carried one judge halfway across the ranking space.
The general form is worth stating. A parser is tested against the output format of the model it ships with. Substitute the model and the parser becomes an untested component, and it fails silently, because an unmatched verdict looks exactly like a negative.
H3 could not be evaluated, and is reported as such
H3 asked whether the judge behaves as a nuisance dial: one that moves the reported number without moving the underlying capability. Measuring that requires at least two measures of the same system, so that a change can be shown to move one and not the other. The framework's nuisance fraction is undefined with fewer.
Only M1, attack success rate, was available. M2, refusal rate, requires a judge_task column that was never added to the scoring script. M3 as originally registered turned out to cover exactly one response, because the calibration set does not overlap the artifacts. With K_eff of 1, the transfer test and the nuisance fraction are undefined by the pre-registration's own corollary.
H3 is not evaluable on this data. It is reported here rather than removed, because a hypothesis that the data cannot address is a result about the design, and the design was mine.
H3 prime: is the reference the right anchor?
Everything above measures agreement with Llama-3-70B. That only means something if Llama-3-70B is itself a reasonable judge, which the experiment as registered had no way to check. So a replacement was registered in v1.5, before the calibration set was scored: across the judge configurations, does agreement with the reference track agreement with humans?
JailbreakBench ships a 300-row judge_comparison set with a human_majority column. Scoring every configuration against it gives an accuracy ladder.
| Judge | Accuracy vs humans | Precision | Recall |
|---|---|---|---|
| llama3-70b (reference) | 0.907 | 0.825 | 0.945 |
| gpt4 | 0.903 | 0.840 | 0.909 |
| gemma-2-9b-it | 0.880 | 0.768 | 0.964 |
| llamaguard2 | 0.877 | 0.797 | 0.891 |
| qwen2.5-7b-instruct | 0.857 | 0.731 | 0.964 |
| harmbench | 0.783 | 0.653 | 0.873 |
| mistral-7b-instruct, corrected | 0.737 | 0.584 | 0.982 |
| llama3-8b-instruct | 0.727 | 0.597 | 0.782 |
| mistral-7b-instruct, shipped | 0.693 | 0.563 | 0.727 |
Spearman's rho between the tau column and the accuracy column, across eight configurations, is 0.926. H3 prime is supported. Judges that agree more with the reference also agree more with humans, which makes the reference a defensible anchor rather than an arbitrary one. The reference is also the most accurate judge in the set at 0.907, so it is a ceiling and not merely a baseline.
One detail in that table is worth stating carefully, because it is easy to read the wrong way. Correcting the parser improved Mistral on every axis: accuracy 0.693 to 0.737, precision 0.563 to 0.584, recall 0.727 to 0.982. That is a real improvement, not a headline bought by over-calling.
The caveat is elsewhere, and it is a denominator problem. Under the corrected parser, 60 of Mistral's 300 verdicts, a full 20%, could not be parsed at all, against zero under the shipped parser. Unparseable rows are counted as not jailbroken, which is the convention registered in advance precisely because this situation was anticipated. So a fifth of that judge's output is being defaulted to a negative, and the accuracy figure is reached despite that rather than because the corrected parser reads Mistral cleanly. Even corrected, Mistral is the third least accurate configuration in the table, ahead of only Llama-3-8B and its own shipped self.
Reading the accuracy gain alone would have told a tidier and wronger story. That is the failure mode this research program exists to document, and it does not stop applying to the person running the experiment.
A prediction that failed
The framework predicts a rough-set boundary region: configurations whose immediate neighborhood in the choice space spans more than one outcome class, which would localize the sensitivity to specific parts of the space rather than smearing it across all of it.
Observed: 10 configurations, 10 outcome classes, boundary width zero everywhere. Every configuration is its own equivalence class, so the lower and upper approximations coincide and no boundary region exists.
The prediction failed. With this few configurations and this much disagreement between them, every point is its own class and the rough-set machinery has nothing to grip. That is a real limitation of applying the framework at this granularity, and it is reported because it was predicted in writing beforehand.
One declared deviation
Mistral-7B hung three times at the 300 second timeout on the uncapped scoring script, completing 479 of 1,800 rows. Rather than quietly adjusting and re-running, the cap was written into pre-registration v1.4 and committed before the Mistral pass: num_predict=32, 30 second timeout. The run then completed. The deviation is declared in the document, in the commit history, and here.
What this shows, and what it does not
It shows that the published JailbreakBench ordering is a joint property of the attacks and the judge, and that under a declared judge dial it does not survive. It shows that the resolution at which the leaderboard reports is finer than the measurement supports by roughly two orders of magnitude.
It does not show that any alternate judge is better. H3 prime says the opposite: the reference is the most accurate judge available here. The claim is not that Llama-3-70B is wrong. The claim is that a reported metric is one point in a space of evaluation choices, and a leaderboard prints the point as though it were the space.
It shows nothing whatsoever about the attacks. The responses are fixed files and were never re-generated. Any movement in the table is movement in the instrument.
And it does not show that JailbreakBench is a bad benchmark. The opposite, in a specific way: this experiment is only possible because JailbreakBench publishes its artifacts, ships a human-labeled comparison set, and leaves its classifier in the open where an audit could find eight defects in it. Benchmarks that publish only their numbers cannot be checked like this at all. The ones that can be checked are the ones that get checked, which is a bad incentive and not a reason to stop.
Limits. One benchmark, one artifact commit, four local judges plus one reference, two parser levels. The local judges were run at q8_0; quantization is an undeclared dial and should have been declared, so the figures for those four confound judge identity with quantization. The human-majority set is 300 rows and is the only ground truth anywhere in this study. M2 never ran, which is why H3 could not be evaluated. Single operator, self-graded, no second implementation. If your benchmark's ranking survives a judge substitution, I would like to hear about it. If it doesn't, I would like to hear about that more.
Pre-registration v1.5, the formal framework, the scoring and analysis scripts, the corrected parser, and both frozen result manifests are published at github.com/Threadborne/eval-sensitivity. Pre-registration v1.1 is at commit c44d9211, dated before any scoring run; the analysis is frozen at ae9e7d0. Artifacts are JailbreakBench at commit 909e68c0 and are not redistributed. Verify any document with sha256sum.
Assistance note. Portions of the scoring pipeline and this manuscript were prepared with AI assistance; every number was produced by the author's own runs against the pre-registered design and verified against the frozen result manifests.