Methodology
How the publication audits agentic systems
The publication writes about how others should audit opaque systems. It would be strange not to be obvious about how we audit them ourselves. These pages publish, in detail, the working methods Black Box Notes uses behind every piece in the archive. They double as a reference for working auditors who want to compare notes.
-
~2400 words
How we audit an agentic stack
The publication's standing walkthrough for an end-to-end audit of an agentic system in production. Six phases, the documents we ask for at each, the failure modes that recur, and the checklist we publish for working auditors.
-
~2100 words
How we read interpretability claims
Distinguishing post-hoc explanation from mechanistic interpretation, decoder visualisations from intervention experiments, and the categories of interpretability claim that survive scrutiny from the categories that do not.
-
~1900 words
How we evaluate transparency reports
What an audit-grade transparency report looks like, what most published reports leave out, and the publication's rubric for scoring a transparency report against the standards a regulator would actually use.
-
~1700 words
Our citation standard
The publication's working rules for citing primary documents, technical papers, regulatory filings, and on-the-record sources. Plus the rules for what does not warrant a citation.
-
~1600 words
How we choose what to cover
The four criteria an event has to satisfy for the publication to commission a piece on it, and the standing reasons we have refused commissions that looked tempting on first read.
Why publish methodology at all?
The publication treats opacity as a property of the relationship between a system and its observer. A publication that writes about opaque systems while keeping its own methods opaque is, by its own definition, part of the problem. So we publish them.
The pages above are working methods, not aspirational ones. They describe what the editorial team actually does — including the parts of the method we are unhappy with, the questions we cannot yet answer, and the cases where our own audit found nothing useful to publish.
Downloadable audit checklists
Every methodology page ends with a downloadable checklist a working auditor can use as a starting point. The checklists are CC-BY licensed — copy them, modify them, ship them inside your own audit reports. We ask for attribution where you can give it. The downloads live in /checklists/.
How these pages get updated
We revise a methodology page when the publication's working method changes. Revisions are dated at the bottom of each page and listed in a changelog there. Material revisions trigger an editor's note pointing to the change.
Tell us what we got wrong
Working auditors who read these pages and find a method incomplete, mis-described, or plain wrong, please write to editors at blackboxnotes dot com. We have built these pages by reading regulatory output, working audit reports, and our own practitioner contributors' notebooks. We have certainly missed something.