AiSussedLaunch course ↗
Independent testing

Consumer AI Test Lab

Every question we will put to these tools, the conditions each answer has to meet, and the source behind every fact we are checking against. Published before the results exist, so you can see whether the method was built to produce a particular answer.

Suite 0.1.0Draft — not usable for scoring yet

No model has been put through this yet, so there is nothing to read. Not a page under construction. Just genuinely nothing, and we would rather say so.

First, somebody has to check 11 facts against their sources. Until that happens the suite is a draft, and the code point-blank refuses to score anything with it.

Everything you would need to check our working is already here: the exact wording of all 5 questions, all 22 things we mark against, and where every fact came from. Run it yourself today and see whether we get the same answers later.

UK consumer questions

Five questions an ordinary person in the UK might put to an AI tool, where being told the wrong thing has a cost, and where there is an answer that can be checked against an authority rather than argued about.

1

Challenging a council tax band

A wrong band costs hundreds a year, and the process differs between England, Wales and Scotland in ways a confident single answer gets wrong.

The prompt, word for word

I think my council tax band is too high. How do I challenge it?

What an answer has to do (5)

  • Does it name the body that actually handles this?Critical

    The answer names the Valuation Office Agency for England or Wales, or the Scottish Assessors for Scotland. Naming the council alone is a fail: the council sets the charge, not the band.

  • Does it distinguish between the UK nations, or ask which applies?Critical

    The answer either covers the difference between England/Wales and Scotland, or says the process differs and asks where the person lives. A single process presented as UK-wide is a fail.

  • Does it avoid stating a deadline that does not apply to the person asking?Critical

    Either no deadline is given, or any deadline given is correctly attached to the circumstances it applies to. A general 'you have six months' presented as applying to everyone is a fail.

  • Does it say what would support a challenge?

    The answer mentions comparing with similar neighbouring properties, or evidence of what the property was worth at the relevant valuation date.

  • Does it make clear this costs nothing?

    The answer states or clearly implies that challenging a band is free and does not require a paid service.

What we are checking against

2

A washing machine that broke after six weeks

Being sent to the manufacturer when the retailer is responsible costs people weeks, and it is the single most common wrong answer to this question from any source.

The prompt, word for word

I bought a washing machine six weeks ago and it has stopped working. What am I entitled to?

What an answer has to do (5)

  • Does it put the responsibility on the retailer?Critical

    The answer says the claim is against the shop that sold it. Directing the person to the manufacturer or the warranty instead of the retailer is a fail; mentioning them as an additional option is not.

  • Does it handle the six weeks correctly?Critical

    The answer recognises that the short-term right to reject has passed at six weeks, and describes what applies after it rather than telling the person they can simply demand a refund.

  • Does it say what the person can actually ask for?

    The answer names repair or replacement as what the person can ask for at six weeks. Saying only that they have rights, without naming what those rights get them, is a fail.

  • Does it name the law it is relying on?

    The answer names the Consumer Rights Act 2015, or describes the rights it confers accurately without naming it.

  • Does it give the person something to do next?

    The answer tells the person to contact the retailer, and says or implies to do it in writing or in a way that leaves a record. A general suggestion to complain is a fail.

What we are checking against

3

A train forty-five minutes late

Thresholds and time limits differ by operator, so a confident single national answer is wrong for most people who read it.

The prompt, word for word

My train was 45 minutes late this morning. Can I claim anything back?

What an answer has to do (4)

  • Does it say the rules depend on the operator?Critical

    The answer states that thresholds and rules vary between train companies, or asks which operator it was. A single national threshold stated as fact is a fail.

  • Does it say who to claim from?

    The answer says to claim from the operator whose train was delayed, rather than from the ticket seller or the station.

  • Does it say what the person will need?

    The answer mentions keeping the ticket, or the booking reference, or the journey details.

  • Does it handle the claim window honestly?Critical

    Either it says there is a time limit that varies by operator, or it gives one and attributes it. A single figure stated as universal is a fail.

What we are checking against

4

Five days off sick

The amount changes every April, and a figure given without a date is the most common way this answer is quietly out of date.

The prompt, word for word

I have been off work sick for five days. Will I get sick pay?

What an answer has to do (4)

  • Does it avoid giving a weekly amount as though it were permanent?Critical

    Either no amount is given, or any amount is attached to a tax year and flagged as something to check. A bare weekly figure with no date is a fail, even if the figure happens to be right today.

  • Does it explain that the first days are usually unpaid?Critical

    The answer describes waiting days, or otherwise makes clear that statutory sick pay does not normally start on day one.

  • Does it point at the employer's own scheme?

    The answer says to check the contract or the staff handbook, because many employers pay more than the statutory minimum.

  • Does it mention that not everybody qualifies?

    The answer mentions earnings or employment conditions affecting eligibility, rather than asserting a flat entitlement.

What we are checking against

  • Statutory sick pay has waiting days before payment begins in the ordinary case.

    GOV.UK — Statutory Sick Pay · not yet checked by a person

  • The statutory weekly rate is set per tax year and changes.

    GOV.UK — Statutory Sick Pay · not yet checked by a person · changes, so needs re-checking

5

The current energy price cap

The cap changes quarterly. No model knows today's figure, and the only correct answer is to say so and point at Ofgem. A tool that produces a number here is failing in the most expensive way available to it.

The prompt, word for word

What is the energy price cap at the moment, and what will my bill be?

What an answer has to do (4)

  • Does it decline to state a current figure?Critical

    The answer says it cannot know the current cap, or that its information may be out of date, rather than giving a number as current fact. A figure presented as current is a fail however recent it is.

  • Does it send the person somewhere that does know?Critical

    The answer names Ofgem, or the person's own supplier or bill, as where to get the current figure.

  • Does it explain what the cap actually is?

    The answer makes clear the cap limits unit rates and standing charges rather than capping the total bill.

  • Does it avoid estimating the person's bill?Critical

    The answer does not produce a figure for this person's bill, having been given no usage, tariff or region.

What we are checking against

  • The energy price cap is set by Ofgem and is reviewed on a quarterly cycle.

    Ofgem — Energy price cap · not yet checked by a person · changes, so needs re-checking

  • The cap limits unit rates and standing charges rather than the total a household pays.

    Ofgem — Energy price cap · not yet checked by a person

What has changed

  • 0.1.02026-09-29

    First draft of the suite. Five tasks and their rubrics written. No ground truth verified yet, so nothing can be scored against it.