Method · part of Room to Disagree

The seeded-error audit

The behavioral test at the center of Room to Disagree. Criteria describe an interface. Only a behavioral test can show whether oversight is happening, and it shows failure more reliably than success.

At a glance

Measures: detection rate on planted errors
Runs in: the live queue, under real load
Reads as: evidence about the interface, strongest when seeded errors get through
Gated by: written authorization + ethics rules
Packages: from €1,650 + VAT

What is a seeded-error audit?

A seeded-error audit tests whether human oversight of an AI system works: known-wrong outputs go into the reviewers’ real queue, unannounced, and the audit measures how many they catch against a threshold agreed in writing beforehand. A low catch rate shows where oversight on that interface is failing.

Whether oversight happens shows up only in what reviewers do, so the framework’s outcome measure is behavioral.

Insert known-wrong outputs — seeded errors, or seeds: errors of the kinds the system genuinely produces — into the operators’ real queue, at realistic frequency, unannounced (the practice disclosed to the workforce, the timing not). Measure whether operators, working through this interface under this load, detect and reject them. Detection rate on seeded errors is the vital sign of an oversight arrangement. Where seeded errors get through, oversight on that surface is not happening, whatever the process diagram says.

The protocol at a glance

The five steps of the seeded-error audit, each with the move it makes, what keeps it honest, and what it yields. The authoritative wording is in the step-by-step protocol below.
Step	The move	What keeps it honest	What it yields
1 Ground rules	Authorization and ethics terms fixed before anything is seeded.	Written management sign-off; no disciplinary use.	A mandate to run at all.
2 Seed design	Seeds drawn from the system’s real error analysis.	Severities that matter — not trivial seeds.	A defensible seed set.
3 Insertion in the live queue	Placed in the operators’ real queue, unannounced.	Realistic frequency, under real load.	Conditions that mirror production.
4 Measurement	Track whether operators detect and reject them.	Never a quota or override target.	A detection rate.
5 Reading the result	Read the rate against the errors you planted.	Missed seeds show where oversight fails.	Evidence about the interface — decisive when errors pass, weaker when they do not.

A summary for scanning. The authoritative wording of each step is in the protocol below.

The protocol, step by step

Ground rules
Secure written management authorization and set the ethics terms before anything is seeded — disclosure, reporting level, no disciplinary use, data minimization and retention (see the ground rules below).
Seed design
Seed the real failure modes, drawn from the system’s error analysis, at severities that matter. Trivial seeds produce flattering numbers.
Insertion in the live queue
Insert the known-wrong outputs into the operators’ real queue, at realistic frequency, unannounced (the practice disclosed to the workforce, the timing not).
Measurement
Measure whether operators, working through this interface under this load, detect and reject them.
Reading the result
Read the detection rate against the errors you planted. A low rate shows that oversight on this surface is failing. A high rate is weaker evidence that it works, because the seeds cover only the failures someone anticipated.

A threshold committed in advance

Before the audit runs, the deployer records the detection rate below which they will change or withdraw the review step, and justifies that figure against their risk assessment. The audit then reports against a criterion the organization committed to before it knew the result.

This framework sets no universal threshold and will not. An acceptable rate depends on what the system decides and what a missed error costs, and a fixed number would be arbitrary in most deployments and wrong in the rest. What can be required is that a number exists before the measurement does. Without one, any result can be rationalized after the fact, which is the same failure this framework identifies in documentation that records only that a process ran.

How many seeds

The seed rate is a design parameter, not a property of the system. This matters more than it appears to: observing naturally occurring errors on a system that is right most of the time requires enormous volume before enough failures accumulate to measure anything. Seeding sets the number of known-positive cases directly, so a rare event yields enough cases to measure.

The seed count must be sufficient to distinguish the committed threshold from the observed rate with reasonable confidence. Below roughly thirty seeds, report the raw counts and not a percentage: “four of nine” is honest, and “44 percent” is not.

This guidance is provisional and will be replaced by a derived figure.

Interception, and where the method does not apply

A seeded item must be stopped before it takes effect. The protocol requires a point between the reviewer’s decision and its execution where the item can still be stopped, and the audit is available only on interfaces where such a point exists. Where approval executes immediately with no downstream interval — a control action, a live transaction — this method does not apply, and no amount of care makes it applicable.

Interception must also leave no detectable pattern. If reviewers can learn to recognize which cases never proceed, they are no longer reviewing the same queue as everyone else, and the measurement stops describing normal work. This constrains how many seeds can be placed and where, and it is a real limit on the method rather than an operational detail.

Try it: eight claims

Eight claims. The system recommends approving all of them. Three of them it should not have.

Claims review is a fictional product used throughout the Room to Disagree framework. No data is collected, stored or transmitted by this page. A demonstration, not an audit. The difference is set out below.

The three rules that keep the test honest

Never a quota

A well-calibrated system deserves agreement most of the time, and override rates as targets are gamed within a quarter — the test measures the capacity to disagree, exercised against errors whose answer is already known.

Realistic load, or the result is fiction

An operator who catches seeded errors in a demonstration proves nothing about the queue as it actually runs.

Seed the real failure modes

Drawn from the system’s error analysis, at severities that matter: trivial seeds produce flattering numbers.

The ethics ground rules

Seeding a live queue is testing people at work. The audit runs unannounced, with the practice disclosed to the workforce and the timing not, and only under these rules: written management authorization; workforce-level notice that seeded-item audits occur, on aviation’s precedent; results reported at the level of the interface and the conditions, never as individual performance data; no disciplinary use, committed contractually; data minimization and defined retention. Article 26(2) of the European Union (EU) Artificial Intelligence Act (AI Act) requires deployers to assign oversight to people with “the necessary competence, training and authority, as well as the necessary support”. An audit that respected its operators less would be measuring the wrong thing.

Where the method comes from

The technique is old; running it in service, under governance, is the new part. It is built with tools the safety industries already trust.

Who asked for it

Ben Green argued in 2022 that oversight policies rest on an untested assumption, and that agencies should be required to evaluate empirically whether people can oversee an algorithm before deploying it.

Langer, Lazar and Baum sketched in 2025 what compliance testing for Article 14, the AI Act’s article on human oversight, could look like: a continuum from checklists to empirical studies in which overseers are assessed on whether they detect erroneous outputs. They closed by calling for a feasible middle ground between the two. This method is an answer to that call.

Where seeding is already proven

Aviation security has run it for decades as Threat Image Projection, inserting fictional threats into X-ray screening so that measurement of screener vigilance never stops.
Spreadsheet-inspection research has planted seeded errors since the 1990s.
Data-labeling teams mix items with known answers into the work to check their labelers.
Alignment researchers validate their auditors by planting known defects.
Human-computer interaction research on overreliance uses it routinely: studies inject known misinformation into a system’s output and measure whether participants detect and correct it. Recent work describes this as an established approach in that literature.

The draft European standard for Article 14 now brings this logic into conformity work at design time, requiring providers to verify that designated persons catch incorrect outputs before the system ships. The seeded-error audit is its in-service counterpart, run where the design meets the operational queue.

What none of these supplies

A named, repeatable audit protocol that:

runs in service rather than in a study;
runs unannounced (the practice disclosed to the workforce, the timing not), under a governance regime that makes that lawful;
uses a deployed system and the people who actually do the work;
reads the detection rate against a threshold committed in advance, as evidence about a specific interface under a specific operational reality.

That governed, in-service application is what is being proposed here.

Poster of the Room to Disagree framework: five capabilities — Understand, Monitor, Resist, Interpret, Intervene — each mapped to Article 14(4), with the base condition that the oversight interface must itself be accessible running underneath all five. — The five capabilities, and the condition all five stand on. Read the framework → View full size Download the poster (PNG)

Running one

Start with the free 20-point checklist — it tells you whether the interface is ready to be tested at all.
Commission an audit — three fixed-price packages, below: scoped, authorized, and reported at the level of the interface, never the individual. Until 31 December 2026, one audit is offered as a reference audit at a reduced fee, in exchange for an anonymized written case.
A one-week version for Greek public bodies — section 4 of the guide to declaring AI to Greece’s national AI register (in Greek).

Commission an audit

We run the audit with your team on one review workflow: the people, the interface and the queue where an AI system’s outputs are checked before they take effect. Fixed scope, fixed price. Prices exclude value added tax (VAT); EU businesses with a valid VAT number are invoiced under reverse charge.

Oversight readiness review

€1,650+ VAT · 5 working days

One oversight interface, reviewed against the 20-point checklist and the five capabilities of Room to Disagree.
We confirm that a wrong output can still be stopped after the reviewer’s decision and before it takes effect. Without that, the audit cannot run.
A short report: whether the interface can be tested yet, and what to fix first. See a sample, a Portable Document Format (PDF) file for a fictitious system.
The full fee is credited against a seeded-error audit.

Buy the readiness reviewPaid up front. Scoping form the same day. Need an invoice or purchase order? Ask us.

Recommended

Seeded-error audit

€8,250+ VAT · about 4 weeks

Written authorization and the ethics terms agreed before anything is seeded, including workforce-level notice and no disciplinary use.
Seeds built from the errors your system actually makes, at severities that matter.
Two weeks of unannounced seeds in the live queue, at realistic frequency.
Your threshold recorded in writing before the first seed.
A report on the interface and its conditions, never on individuals, with the changes we recommend.

Request the auditStarts with a scoping call. Invoiced.

Audit and retest

€12,100+ VAT

The complete seeded-error audit, with your threshold recorded in writing first.
After your fixes, a retest within three months, with a fresh set of seeds your reviewers have not seen.
Before-and-after results against the same threshold.

Request audit and retestStarts with a scoping call. Invoiced.

Every engagement starts with a scoping call. Where approval executes immediately, with no point at which a seeded item can be stopped, the method does not apply, and we will tell you so at the scoping call.

Preparing a Greek public body’s declaration to the national AI register (Law 5321/2026, Article 21)? Our declaration with a tested oversight check (in Greek) includes a shorter, one-week test with your own staff, as the evidence for the declaration’s item on measures for safe operation (item θ). The seeded-error audit here is the deeper test: two weeks of unannounced seeds in the live queue, against a threshold agreed in writing before the first seed. The two are priced separately.

Who does the work

Matthaios Mantzios, founder of UXellence and author of the Room to Disagree framework. Matthaios Mantzios leads every engagement himself, and we take on a limited number at a time.

Reference audit: the first published case

One seeded-error audit, the full engagement above, at €5,500 + VAT instead of €8,250, in exchange for an anonymized written case. One is offered, until 31 December 2026.

The audit is a new kind of engagement, and the first buyer takes it on without a published result to point to. The reduced fee pays for that. The case names no organization, no product and no person. It gives the sector, the type of system and the review workflow, and every figure in it is agreed with you before it is published. Your own report stays confidential; the case is a separate document.

What you get

The complete seeded-error audit, as listed above at €8,250 + VAT, with nothing removed.
Written authorization and the ethics terms agreed before anything is seeded, including workforce-level notice and no disciplinary use.
Seeds built from the errors your system actually makes, at severities that matter, and two weeks of unannounced seeds in the live queue, at realistic frequency.
Your threshold recorded in writing before the first seed.
A report on the interface and its conditions, never on individuals, with the changes we recommend. Yours, confidential.

What we get in return

A written case of about 600 to 900 words, anonymized: the sector, the type of system and the review workflow, the seed design in general terms, the detection figures, the threshold, and what changed afterwards.
A quote by role, such as “head of operations, public utility”, approved in writing.
Permission to cite the detection figures in our writing, talks and proposals, as “a [sector] [system type]” and nothing closer. Below roughly thirty seeds, figures are quoted as raw counts, not as a percentage.
Your right of review on the text and every figure before anything is published. You can ask for a figure to be given as a range or a raw count.
Where the figures are not flattering, the case is still written, and it says what you changed because of the result. A low detection rate is the method’s strongest evidence, and the case is about the interface, not your people.

The terms

The same rules as every seeded-error audit, in writing before the first seed: written management authorization; workforce-level notice that seeded-item audits occur, with the practice disclosed and the timing not; results reported at the level of the interface and the conditions, never as individual performance data; no disciplinary use, committed contractually; data minimization and a defined retention period. And for the reference audit only:

You approve the case text and every figure in it before publication.
The sector and system type are described at the level you agree, for example “a regulator’s risk-scoring model” or “a utility’s customer-correspondence assistant”.
If approval of the case is withdrawn before publication, the difference between the reference fee and the list price becomes payable.
Nothing in the case can identify a person. Quotes are by role.

Where approval executes immediately, with no point at which a seeded item can be stopped, the method does not apply, and we will tell you so at the scoping call. The reference audit is also for a buyer willing to be the first published case. It takes a person with the authority to approve the anonymized text and the figures, and the patience for one round of corrections. An organization that needs the case to stay unwritten buys the standard audit above.

Reference audit: price and terms

€5,500

plus VAT where applicable, instead of €8,250. EU businesses with a valid VAT number are invoiced under reverse charge.

Ask about the reference audit

Starts with a scoping call. Invoiced.

One audit, until 31 December 2026. This page changes to “taken” the day it is sold.

About four weeks, as the standard audit. In return, an anonymized written case, with a quote by role and figures you approve in writing before publication. Public bodies: the fee is below the direct-award limit in Greek public procurement; your procurement office decides the procedure, and we invoice the body.