The pilot
Twenty to thirty people in Britain. One real thing each, on this site, while somebody watches and says as little as possible. The method is below, frozen, and it went up before anybody was asked to take part.
Why this is public
Pick the measures after the sessions and you will find out that the thing you built is good. Everybody who has run one that way came out pleased.
So the measures are here, dated, before anybody has been asked to take part. If they change, the change will be appended underneath with a reason rather than tidied into the original.
It is the same shape as anything on the Community Experiments page. The rule is the same rule, and it should not get easier when the work is ours.
What gets measured
- Task success
Whether you got the thing you came for, said in your words at the end rather than ours from watching. It is the rule the missions page follows: finishing the steps is not the same as it having worked.
Not: A percentage. Twenty people is not a sample and a percentage of twenty is a number that invites being quoted.
- Comprehension
You read one explainer, then say back what it means. And what it would change about something you actually do. Written down as you said it, not scored.
Not: A quiz. A quiz measures whether somebody can pick the right option, which several of these pages teach you is a different skill from knowing.
- Confidence
One question, before and after, in the same words both times. How sure are you that you would spot a wrong answer? Kept as a pair, with whatever you said about why.
Not: Evidence that the site works. Confidence going up in somebody still wrong is the outcome this whole site exists to avoid, so the pair is read next to what they understood.
- Support needs
Everything you needed that was not on the screen. A word explained, a step we assumed, something you went looking for and could not find. And every point where you had to ask.
Not: A backlog. The interesting ones are the questions nobody thought to ask, and those only turn up if the facilitator stays quiet long enough.
Who has to be in the room
Numbers rather than the word “diverse”, so it either happened or it did not. If we cannot fill the room, the write-up says who came and that it does not generalise.
At least 6 People outside England
Housing law, deposit schemes, fraud reporting and council tax all differ. Nobody in Scotland, Wales or Northern Ireland has ever told us what we got wrong.
At least 4 People who use a screen reader, magnification, voice control or a switch
The largest gap on the accessibility page. No automated check will ever close it.
At least 6 People who have never deliberately used an AI tool
The whole site claims to be for them. So far it has only been read by people who were already interested.
At least 4 People over 65
Scams, letters and benefits land on them more than on anybody. None of what we wrote about it has been read by them.
At least 6 People on a phone, on mobile data, as their only device
A machine checks the layout at 375px on a fast connection. That is not the same thing at all.
The method
Written on 2026-09-30. Not running — nothing has started, so nothing is frozen yet.
What is being compared
Not two things against each other. Twenty to thirty people, one real task each, brought with them. Somebody watches and says as little as possible.
How it is run
- You bring a real thing you were going to do anyway. A letter to write, a deal to understand, a scam to check. Nothing is invented for the session.
- You say what you would do about it today, before seeing the site at all. That gives the afternoon something to be compared with.
- You use the site on your own device and your own connection. The facilitator is on a call, not over your shoulder.
- The facilitator helps only if you have been stuck two minutes and ask. Every time they do, it is written down as a support need.
- Confidence is asked before and after, in the same words both times.
- At the end you say whether you got what you came for, and that answer is kept exactly as you gave it.
- Nothing from your own documents is recorded. If the task needs a real letter, it stays on your screen.
What counts as right
You say you got what you came for, and you can say what you would do next. Both, because either on its own has been managed by every confident wrong answer ever given.
What would mean we found nothing
Fewer than twenty people finish a session, or the groups above are not filled. Then it says nothing about anybody except the people who came. We report who turned up, what happened, and that it does not generalise, rather than drawing a line through twelve points.
Who checks
Two people read every session note separately. They have to agree what counted as a support need before any of it is summarised, and where they disagree both readings go up.
What this cannot show, whatever it finds
- Whether the site works for anybody who did not volunteer. People who agree to a session about AI are already the sort who agree to a session about AI, and no amount of recruiting fixes that.
- Whether confidence that went up was earned. It can be read alongside comprehension, and that is a judgement rather than a measurement.
- Anything about how the site performs over months. One afternoon is one afternoon.
- Whether a screen reader user can use this site. Four sessions finds the barriers obvious to four people. That is a floor, not a verdict.
What we found
Nobody has been recruited and nothing has been run, so there is nothing here.
The method above was written and frozen first, which is the whole point of putting it up before there is anything to report. When there is a result it goes here, including if it is that we could not fill the room.