How we evaluate transparency reports

A transparency report is a public document by which an operator of an opaque system attempts to make the system less opaque. The publication reads a dozen of these a quarter. This page describes how.

The category problem

"Transparency report" covers at least four different kinds of document. We find it useful to keep the categories distinct:

Statutory disclosures. Reports an operator publishes because the law requires it (financial disclosures, EU AI Act conformity assessments, sectoral regulator filings). The content is constrained by statute; the question is whether it covers what the statute actually requires.
Sector-standard disclosures. Reports an operator publishes against an industry framework (ISO/IEC 42001, NIST AI RMF, third-party AI ethics frameworks). The content is constrained by the framework; the question is whether the report meets the framework or merely cites it.
Voluntary public disclosures. Reports an operator publishes for reputational reasons in the absence of a binding requirement. The content is unconstrained; the question is what the operator chose to include and what they chose to leave out.
Audit-grade transparency reports. Reports written or co-written by an independent auditor against a scoping engagement that defines the questions. The content is constrained by the audit scope; the question is whether the scope was adequate to the claims.

The publication's six-axis rubric

We score a transparency report on six axes. Each axis runs from 0 (absent) to 3 (audit-grade). Composite scores are not the goal; the value of the rubric is the per-axis reading it forces.

1. Scope clarity

0: the report does not state what system, deployment, or time window it covers.
1: the report names the system but not the deployment scope.
2: the report names the system, the deployment scope, and the time window.
3: as 2, plus a written rationale for excluding adjacent systems.

2. Quantitative content

0: no numbers, or numbers without methodology.
1: numbers reported with methodology described in summary.
2: numbers reported with methodology described in full, with the underlying definitions.
3: as 2, plus the raw data or a defended summary of it, plus an independent-replication path.

3. Failure-mode disclosure

0: no failure modes disclosed.
1: failure modes described in general categories.
2: specific recent failure modes named with the conditions that produced them.
3: as 2, plus the operator's recovery actions and the residual risk.

4. Independence of authorship

0: written entirely by the operator's communications function.
1: written by the operator's engineering or risk function.
2: co-written with a named independent auditor under a written engagement.
3: written by an independent auditor against a published scope, with the operator's response as an annex.

5. Counterfactual disclosure

0: the report does not discuss what it has chosen not to include.
1: the report names what it has excluded.
2: the report names what it has excluded and why.
3: as 2, plus the standing reason a future report will or will not extend coverage to the excluded material.

6. Affordance for reader response

0: no mechanism for a reader to dispute, correct, or follow up.
1: a generic contact channel.
2: a named editor or audit lead the reader can write to with documented response windows.
3: as 2, plus a published process for the operator's revision of the report based on substantive reader response.

What most published reports score

Across the publication's running review set, the most common pattern in the voluntary public disclosures category is high marks on axis 2 (quantitative content), middling marks on axes 1 and 5 (scope and counterfactual disclosure), and very low marks on axes 3 and 4 (failure-mode disclosure and independent authorship). The composite tells less than the per-axis reading: a report that scores 3-3-0-0-0-1 is a report whose quantitative claims may be reliable but whose strategic omissions are the story.

What audit-grade reports do differently

They include the audit scope as a published annex.
They name the auditor's findings before the operator's response.
They retain the auditor's language on disputed findings.
They publish on the auditor's letterhead, not the operator's.
They include the auditor's recommendations the operator has chosen not to act on, with the operator's reasons.

The "disclosure inversion"

A pattern the publication watches for: a transparency report that publishes ostentatious detail on the matters that flatter the operator and aggregates over the matters that do not. The aggregation is the disclosure. A report that says "our model has an X% accuracy on benchmark Y" while declining to characterise the model's behaviour on the long tail of production inputs is reporting the controllable and omitting the consequential.

What we will not call a transparency report

We will not use the phrase "transparency report" for documents that do not contain at least one substantive disclosure the operator was not commercially obliged to publish. A document containing only the disclosures the law requires is a regulatory filing. We call it that.

How we cite transparency reports

Citations include the publishing entity, the report title, the date of publication, and the URL of the underlying document (not the press release announcing it). Where the report has been revised since publication, the citation includes the revision date the publication retrieved.

Changelog

2026-05-22. Initial publication.