How we know things

• Every question is printed word for word, so you can put it to a model yourself and see what you get. A score you cannot reproduce is just an opinion wearing a number.
• Every check is a yes or a no, with the condition spelled out. Nothing is marked out of five, because no two people ever agree on what a four looks like.
• Every fact we mark against names its source, and a person has to go and check it before the suite can be used at all.
• A person reads every result before it goes up, and anything marked wrong has to say why in plain words.
• Results are tied to a suite version and an exact model name. We never compare across versions, and “Claude” is not a version any more than “car” is a model of car.
• No model gets an overall score. Deciding that a parking question is worth more than an energy one is a judgement we could not defend, so results stay per question.
• Results we throw out stay up, marked as thrown out. If you cannot see the correction, it is not a correction. It is a delete button with better manners.

• Everything is sorted by what it costs you to miss it. Not by when it happened, and definitely not by how excited everyone else got.
• Most entries say there is nothing to do, in those words. A radar where everything is urgent gets muted within a fortnight. Then the one that mattered goes unread with all the rest.
• A new model turning up is filed under nothing to do. Nothing you own stops working when one arrives. We have not tested it, or the one you already use, so we are hardly going to tell you to switch.
• Every entry says where we saw it. If we have not managed to look lately, you get told that too. An empty page would let you assume all is quiet, which is a different thing entirely.

• What they say is their words, their link, and the date we read it. We are not nodding along. We are writing it down, so you can hold it against whatever they say next year.
• What we saw only turns up once a person has gone and looked. Everything we mean to check and have not yet is listed underneath. You can see the homework we still owe you.
• What we think is ours, and it has to show its working. If a view rests on nothing but a company's own account of itself, we say so. That is much weaker than it sounds.
• None of this tells you whether a model is any good. We have not tested one yet. What you get for now is how each company writes things down — a smaller question, and one we can actually answer.

• Nothing here calls an AI model. Asking one whether you should use AI is like asking a barber whether you need a haircut, except the barber would at least give you the same answer twice.
• Every rule below is ours, written down, and the same every time. We show you which ones applied to your answers and which did not, so you can argue with a specific rule rather than with a verdict.
• No score. A number out of ten sticks in your head and the reasons fall straight out of it, which is exactly the wrong way round.

• You cannot reliably tell. Not with practice, not with a good eye, and not because you are the sort of person who is careful. For a competent fake of somebody specific, an ordinary person's success rate is not meaningfully better than a coin toss, and every year it gets worse rather than better.
• Neither can we, and neither can the tools that say they can. Detector software gives a probability, and a probability is exactly no use in the only moment that matters, which is when somebody is asking you to send money in the next twenty minutes.
• That sounds like bad news and is not. Every drill here has a safe answer that works identically whether the thing was a fake or your actual sister, and none of them requires you to be right about which it was.

• Every objection has to point at one of our own explainers, and we check the page is really there. That means the problem is real. It does not mean it is yours.
• Whether any of it applies to what you are actually doing is the whole question, and nothing on this page can answer it. That is what the experiment is for. Fifteen minutes beats an afternoon of arguing.
• The case for is the same length as the case against, on purpose. Anything can be gloomy on demand. Gloom is not the same as care, and it is no harder to produce than enthusiasm.

• These are worked examples. Nobody stood over anyone with a stopwatch, and the minutes are our best guess at how these jobs really go. Take them as the shape of the trade-off, not the size of it.
• The checking time is left in, which is the entire point. Most comparisons time the human route honestly, time the AI route up to the moment a draft appears, then stop. That is how an hour of work turns into a ninety per cent saving.
• Your own number is the only one that means anything. Use one of the tools here and we will ask you afterwards how long you think it saved you. That figure is yours, you can change it, and we never work it out for you.

A result says how a model answered 5 questions, on one particular day. If something happens that could change its answers, the result goes stale. If it could not, it does not. Stale results stay up with a label on them, because hiding one would lose the record, and the record is the whole point.

• retiredNo retest

  There is nothing left to test. The result stays as a record of something that existed, marked as no longer current.

• deprecatedNo retest

  Nothing about how it answers has changed. What has changed is how long it will be there, and that belongs on the result rather than in a retest.

• capability lostRun the suite again

  Something it used to do it no longer does. Any answer that leaned on that is now wrong, and we cannot tell from here which ones did.

• context smallerRun the suite again

  It holds less at once, so anything that used to fit may now be quietly truncated. A result produced when the whole question fitted is not a result about the model as it is now.

• price upNo retest

  What it costs cannot alter a word of what it says. Testing it again would tell us only what we already knew, at our own expense.

• left previewRun the suite again

  A preview and a settled release are not the same thing to test. Results from the preview describe something the maker has since said was not finished.

• capability gainedRun the suite again

  A model that can do something new may answer differently on questions that have nothing obviously to do with the new thing. That is not intuitive, and it is why the trigger is not narrower.

• context biggerRun the suite again

  A bigger window usually comes with other changes, and we cannot see which. The cheap assumption is that the answers may differ.

• renamedNo retest

  The same thing is being called something else. The identifier on the result has to follow, but re-running the suite would produce the same answers from the same model.

• price downNo retest

  The same as a rise, for the same reason. What a model costs is not part of what it says, and a cheaper model gives the answer it already gave.

• new modelRun the suite again

  Nothing is stale, because nothing is known. Until the suite has been run against it, we have no basis for saying anything about it at all.

• Nothing here gets painted over. A change writes a new version and keeps the old one, dated, with where it came from — so what we believed in March is still sitting there in September, looking sheepish.
• A correction comes with a note from whoever made it, saying what was wrong. A version that quietly changed with no explanation is not a correction. It is just a change, and it reads like one.
• A test result we threw out stays up, marked as thrown out. If you cannot see the correction, it is not a correction. It is a delete button with better manners.
• Change a question or the marking, and the whole suite gets a new version number. We never compare results across versions, because they are answers to different questions.

Empty so far. Not because we are careful, but because nobody has caught us out yet. The list is here already rather than appearing after the first mistake, which would be a very convenient moment to start keeping one.