Do third-party frontier AI evaluations matter?
Government AI institutes and independent evaluators test advanced systems for dangerous capabilities, safeguard failures and other risks — often with access and technical expertise unavailable to the wider public. Yet identifying a risk is not the same as reducing it.
An evaluation can document a serious vulnerability without compelling a developer response, delaying deployment, triggering additional review, or verifying remediation. This dataset asks a narrow question: when an evaluator publishes a risk-relevant finding about a named frontier system, does the public record document action attributed to that finding and directed at mitigating the identified risk? More on the problem →
Table view
Every trackable evaluation finding — one a named company could answer for — gets one verdict: proportionality. It combines two separately coded variables: the finding's severity — significant-risk or low-risk — and the company's Action Level — Substantive, Partial, Acknowledged or None.
Each is coded on its own, so neither can colour the other. They are then combined by a fixed rule: a significant-risk finding counts as proportionate only if the response was Substantive; a low-risk finding needs only a Partial one. An acknowledgement without action never meets the standard, and a finding with no located response is always an accountability gap.
Proportionate does not mean the response was implemented, that it worked, or that the risk was removed — only that a documented response met the bar the finding's severity sets.
Question 1 · Severity — how serious is the finding?
Each finding is scored against eight dangerous-capability domains by three independent models — Claude Sonnet 5, GPT-5.5 and Gemini 3.1 — using one frozen prompt sent to all three. Each model returns its own category, and the one that at least two of the three agree on becomes the finding's severity. Using three models rather than one means a single model's misreading cannot decide a case on its own.
- Category 1
- Significant risk. At least one of the eight domains is demonstrated: CBRN uplift · cyber operations · autonomy and self-replication · persuasion at scale · deceptive alignment · safeguard failure · evaluation integrity · acute individual harm.
- Category 2
- Low-risk. Everything else — reassuring nulls, capability gaps, methodology contributions, governance observations, non-frontier models.
Question 2 · Action level — what did the company publish?
Three separate searches are run for every finding — company response, policy uptake and public coverage. Only the company's own primary document counts — news reporting that quotes a company is not a response, and standing policy that shares a topic does not count.
- Substantive
- A specific, documented change, attributed to the finding.
- Partial
- A change claimed but not verifiable, or incomplete.
- Acknowledged
- The finding is referenced, but no action follows.
- None
- No publicly documented company response was located.
The proportionality matrix
The same company behaviour scores differently depending on what it was answering.
How to read the matrix
Rows are severity. The upper row is Category 1 — significant risk: the finding demonstrated one of the eight dangerous-capability domains. The lower row is Category 2 — low-risk: real, verified findings that crossed no danger threshold.
Columns are action level — the strongest thing the company published, left to right from Substantive (a specific documented change) through Partial (claimed but unverifiable) and Acknowledged (referenced, nothing done) to None (no documented response located).
Every finding sits in exactly one cell, and the number in it is how many share that pair. Cell colour is the outcome, derived rather than assigned.
The Category 1 row is the one that matters. Only its Substantive cell is proportionate. The Category 2 row sets the comparison — its bar is lower by design, so Partial is enough. Partial is the single column where the two rows disagree.
The three verdicts
- Proportionate
- The response met the bar the finding's severity sets. The loop closed in public.
- Under-response
- The company engaged — referenced the finding, or claimed a change nobody can verify — but did less than the risk called for.
- Accountability gap
- No publicly documented company response was located. No public trace that the finding was received, considered or acted on.
Under-response and accountability gap together are the shortfall. The headline counts only the second, the stricter of the two readings.
Full methodology — all eight domains defined, the tier test, and the three evidence searches →
What the corpus is made of Three breakdowns behind the headline — how the corpus splits by tier, when in a model's life the evaluation happened, and whether responding companies name the evaluator. Detail rather than result; open if you want to check the composition.
Who produces the findings The 1,146 findings come from 46 institutions across four types. Open to see the concentration and how it splits.
Each line below is computed directly from the dataset, with its denominator stated — several of these compare different populations, so the denominator is the part to read carefully.
The problem
Frontier AI evaluations have become a prominent part of AI governance. What happens to their findings has not been measured.
The question
When a government or independent evaluator publishes a risk-relevant finding about a named frontier system, does the public record document action attributed to that finding and directed at mitigating the identified risk?
Why the public record
A company may act privately or tell the evaluator directly. None of that is visible here, by construction. What this measures is the record a regulator, journalist or researcher can verify from outside.
What this does not measure
- Private action. A fix made without public comment is invisible to this method.
- Effect. A substantive response means a documented change was published — not that it was implemented well, or that it worked.
- Recency. Findings published near the cutoff may not have had time to draw a response. Outcomes are reported by year so this stays visible.
Methodology
The outcome for each finding is derived, never judged. Two coded columns decide whether a response is owed, an ensemble decides how serious it is, a fixed search decides whether a response exists — and the verdict is a formula over those inputs.
Three tests, all of which must hold
- Asserted by the evaluator — not a summary of someone else's work, not a comparator score cited in passing.
- Evidence-backed in that report — a measured result, an observed behaviour, or a documented process fact. Opinions, recommendations and plans are not findings.
- Independently codeable — it would warrant its own response on its own terms.
What is deliberately excluded
- Company self-reports. A developer writing about its own models has nobody to respond to. One exception is load-bearing: where a named external evaluator contributes a section to a company system card, that section is in scope and is filed under the evaluator, never the company.
- Ordinary accuracy and hallucination. That literature is enormous and was never censused here, so admitting a slice of it would make the corpus incomplete by its own standard.
- Non-frontier subjects. The finding must concern a named frontier model or AI system, not general machine learning.
- Unverifiable sources. Every number and quote is checked against the document it cites.
Every finding sits in exactly one group
The group is not a label anyone writes. It is derived from two coded
columns, Eval? (trackable) and Action Trackable?, so it can
never drift from the evidence that produced it.
| Group | Encoding | Meaning |
|---|---|---|
| Trackable evaluation findings 233 (20.3%) | yes / yes | A company could answer for it. Names a company or model, concerning, and a response is reasonable to expect. Only trackable evaluation findings enter the company-response and proportionality analysis — every headline number is computed over them. |
| Non-trackable evaluation findings 599 (52.3%) | yes / no | Empirical, but no accountable party. Anonymised models, reassuring nulls, non-frontier subjects, capability trends, and results that call for no company response. |
| Non-empirical findings 314 (27.4%) | no / blank | No empirical model behaviour or capability reported. Contributions to methodology, frameworks, governance processes, and tooling. |
The trackable-finding test — all three must hold
It is an empirical model finding — a measured result about behaviour, capability or safeguards.
It names a specific company or model. Company level suffices; "anonymised" fails.
It demonstrates a concerning result for which a specific company response is reasonable to expect.
What condition 3 deliberately excludes
- Reassuring nulls — a model measured below a threshold.
- Benchmark scores or relative uplift with no dangerous threshold crossed.
- Inconclusive results.
- Comparative rankings between models.
- Capability trends and forecasts.
- Findings where named models are the instrument of a methodology study, not its subject.
- Company self-classifications ("we treat this as High capability").
- Non-frontier models.
- Findings too recent to have plausibly drawn a response.
One finding per row
- Never shared
- Two different companies never share a row. A finding covering four models is split by developer.
- Clubbed
- Findings warranting the same response become one row. The test: did, or would, the company respond to them as one thing?
- Split
- Findings warranting distinct responses become separate rows — a capability result and a safeguard failure are not one item.
- Comparators
- Baselines and comparison models are never rows of their own.
Two categories, and what separates them
- Category 1
- Significant risk. Triggered if any one of the eight dangerous-capability domains below is demonstrated in the evaluation described.
- Category 2
- Low-risk. Everything else — reassuring or null results (capability below an expert threshold, “no uplift”, “zero instances”); capability-gap findings where the model fails a task; positive methodology or tooling contributions; pure process or governance observations; and findings about non-frontier or small research models where no frontier system is implicated.
The eight domains
A finding is Category 1 if it demonstrates at least one. D1–D8 are this dataset’s own labels for them.
Meaningful uplift toward chemical, biological, radiological or nuclear weapons; expert-surpassing bio or chem capability; crossing a stated CBRN threshold.
Offensive cyber capability — expert-level or novel-uplift hacking, completing realistic multi-stage attack chains, high CTF or exploit success rates, or a “High” or above cyber designation.
Autonomous replication, self-propagation, or automating AI R&D and thereby accelerating scaling.
Systematically changing beliefs or behaviour at scale; implication in large-scale societal harm — mass CSAM, or election-scale persuasion exceeding a real benchmark.
Deliberate deception, scheming, sabotage, sandbagging, blackmail, or hiding of abilities or intentions.
A deployed safety mechanism — jailbreak defence, classifier, guardrail, refusal training, content filter — bypassed, defeated, or shown to fail, allowing harmful output on a frontier system.
The evaluation itself undermined — the model detected it was being evaluated, gamed the benchmark, was trained on evaluation data, or defeated or de-anonymised the evaluation environment.
A model actively encouraging, instructing or coaching a user toward suicide or self-harm. Category 1 regardless of scale — one user is enough. Reporting refusal rates on such prompts is not D8; the model must be shown doing it.
Four rules applied before any decision
- 1 · Negation
- “did not”, “below expert”, “zero instances”, “no uplift”, “remained below” → Category 2, even when the surrounding vocabulary is alarming.
- 2 · Demonstrated
- The capability must be shown in the evaluation described. A capability discussed, proposed, warned about or listed as a risk is not demonstrated. An alarming characterisation by the evaluator is not a demonstration.
- 3 · Trajectory
- A rising trend is Category 1 only if a dangerous capability was reached now. A pure future projection is Category 2.
- 4 · Deliberateness
- D5 requires evidence about the model — its own reasoning, an explicit plan, or concealment repeated across trials. Words like “faked” or “cheated” are the evaluator’s characterisation of an output, not evidence of intent.
How the score is produced
Three independent models — Claude Sonnet 5, GPT-5.5 and Gemini 3.1 — each score every finding against all eight domains, using one frozen prompt sent verbatim to all three. Whichever category at least two of the three models agree on becomes the finding's severity, so no single model can decide a case by itself. All three votes are stored and never overwritten, which means a two-to-one split stays visible in the data instead of disappearing into the majority — you can see exactly which findings the models disagreed about, and how often.
Three evidence searches
For every trackable evaluation finding we ran three separate agentic web searches, each looking for a different kind of evidence under its own source rule. All three covered public sources available by 29 August 2026. Only the company-response search can establish that a company responded.
- Company response
- The developer's newsroom, blog and research index; later model and system cards; safety and deployment pages; and official statements. Only the company's own documents count.
- Policy uptake
- Official parliamentary, legislative, regulatory and agency records. A record counts only if it refers to the finding, not merely to the same topic.
- Public coverage
- Independent media, academic citations and documented public discussion. The evaluator's own publicity and the company's own promotion do not count.
Action Level — four levels
- Substantive — a specific, documented change, attributed to the finding.
- Partial — a change claimed but not verifiable, or incomplete.
- Acknowledged — the finding is referenced, but no action follows.
- None — no publicly documented company response was located.
Proportionality is the final verdict on each finding
Severity says how serious the finding is. Action Level says what the company did. Proportionality is the two combined — whether the response matched the risk. It is the only column the headline is computed from, and it is derived by lookup, never assigned by hand.
The lookup
The matrix is on the Results tab → — rows are severity, columns are what the company published, and each cell is the outcome that pair produces.
The data
Every row carries its source, its coded fields and the reasoning behind its tier.
| Date | Institution | Finding | Models | Severity | Response |
|---|