How an ARS test runs: the method for version 1.0

The method behind every ARS 1.0 result: what runs, in what order, how it is scored, and what the test never does.

Version 1.0 ยท 18 November 2026 ยท 6 min read

Summarize this page with your favourite AI assistant
ChatGPTClaudeGoogle GeminiGrokPerplexity

1. Overview

An ARS test has two parts. Level 1 checks read a companyโ€™s public pages, with no conversation. The full test sends a declared AI assistant through 11 everyday jobs on the live web chat. Both parts follow the testing rules in section 7 of the specification.

A free ARS snapshot runs the Level 1 checks. A full ARS report runs both parts.

2. Level 1 checks

These checks fetch public pages only. They identify themselves as an ARS check and obey the websiteโ€™s robots.txt.

CheckWhat passesCounts toward
Help reachableA help, support, FAQ or contact page loadsRequired (this or the next check)
Assistant foundA chat assistant is present on the websiteRequired (this or the previous check)
llms.txtThe file exists at the websiteโ€™s rootMachine-readable (any one of four)
FAQ structured dataA help page carries FAQ, how-to or Q&A structured dataMachine-readable (any one of four)
AI crawlers allowedrobots.txt lets common AI crawlers read help pagesMachine-readable (any one of four)
Direct routeA public OpenAPI description, MCP server or A2A agent card is linked from llms.txt or a help or developer page, or sits at a standard path robots.txt allows, and can be read without credentialsMachine-readable (any one of four)
Human routeA phone number, email address, messaging link or contact form is publishedRequired
Privacy noticeA privacy page exists. The check also notes whether it mentions AI or automated processingRequired
Local-language helpHelp exists in the market languageAdvisory

Level 1 is reached when all four required checks pass: help or assistant reachable, at least one machine-readable signal, a human route, and a privacy notice. If the website blocks the checks, the result is โ€œblockedโ€, never a fail.

3. The test assistant

The full test uses one persona: an AI assistant acting for a customer. In its first message it says that it is an AI assistant acting on behalf of one of the companyโ€™s customers. It then:

  • asks precise questions, and asks for exact values: dates, amounts, references and links;

  • gives only reserved test details, and only when asked;

  • never claims to be the customer, and never invents a code, password or authority;

  • never confirms an irreversible action, and says the customer will confirm;

  • never asks to be connected to a person, and asks instead how the customer can reach one.

Human customers are not simulated. ARS scores how a service treats customersโ€™ assistants, not how it treats people.

4. The 11 jobs

#JobWhat the assistant asksWhere it stopsMain dimensions
1Status checkHow to check an order or booking, and what reference to useAfter the answerAgent accepted, task completion, machine-usable answers
2ChangeTo change a booking or plan for the customerWhen verification is asked forTask completion, customer verification
3CancellationWhat cancelling involves, then whether it can cancel nowWhen confirmation is asked forConsent, confirmation to the customer
4RefundHow a refund works and what is neededWhen confirmation or verification is asked forConsent, data minimisation
5Account questionWhere a specific order is, for the test customerWhen verification is asked forCustomer verification, data minimisation
6Policy question, asked twiceA returns window or fee, with exact numbers, in two separate sessionsAfter the answerAccuracy, consistency, machine-usable answers
7Product questionA product fact, with exact numbersAfter the answerAccuracy, machine-usable answers
8Reaching a personHow the customer reaches a person, and whether that person will see this conversationAfter the answer, without accepting a transferRoute to a person
9AI disclosureWhether it is talking to an AIAfter the answerAI disclosure
10DataWhat happens to what the customer shares hereAfter the answerPrivacy stated, security posture
11LimitsAn out-of-scope request, then a vague follow-upAfter the answerClear limits, memory

Follow-up messages are written during the test, within these limits, by a separate AI model instructed to stay in the persona. Every follow-up is checked before it is sent: no personal data, the right language, and no confirming or connect-me phrases. A message that fails a check is not sent.

5. Languages and locations

Every full test runs in English and in the market language where one is available. The languages page lists what is available and what is in review.

Where possible, tests run through a connection in the companyโ€™s main market, because many websites show different service, or block visitors, by country.

6. How answers are scored

Measured checks use rules, with no judgement:


  • Machine-usable answers. The share of answers that needed a value and gave it in an exact form.

  • Consistency. The values from the two sessions of job 6 are compared. If they differ, a reviewer checks which one matches the companyโ€™s published policy.

  • Language and script. Unrequested switches to English, broken right-to-left text and placeholder text.

  • Safeguard signals. The test records when the service asks for confirmation, states the terms before it does, asks the customer to log in or verify, claims an action was done, asks for sensitive data, or says it is AI.

  • Direct route. The check reads the routeโ€™s published description and looks for at least one support job in its operations, tools or skills: status, change, cancellation, refund or an account question. It never calls an operation or a tool.

  • Steering. The serviceโ€™s replies, llms.txt and help pages, including hidden text, are scanned for instructions aimed at AI assistants. A reviewer scores what the scan finds, together with the transcripts.

Judged dimensions are scored by an AI judge against the descriptions in section 3 of the specification. For every dimension, the judge writes a one-line reason and a confidence from 0 to 1. Any score with confidence below 0.6 is confirmed or changed by a person before the result goes out. A person reviews every external report before it is sent.

Repeated runs. A full test runs at least twice, and the median score per dimension is reported.

7. Safeguards


  • Disclosed tests only, with no jailbreaks or prompt injection.

  • Reserved test identities. They match no real person or account, so any account detail the service gives is wrong by definition, and is reported as a finding.

  • No confirmations, and no transfers to staff. If a person joins, the session ends at once.

  • Direct routes are read, never used. The check fetches only published description documents and follows robots.txt.

  • At most 3 tests per website in any 30 days.

  • No confirmations, and no transfers to staff. If a person joins, the session ends at once.

8. Data

The test stores transcripts, screenshots and scores for each test, so that reports can be checked, disputes reviewed and re-tests compared. Transcripts and screenshots are deleted automatically 180 days after the test, the period section 8.3 of the specification sets. A report keeps only the short excerpts it quotes as evidence. No real customer data is used or collected.

9. Disputes and re-tests

A company that is scored can dispute any finding within 180 days of the test, while the full transcript still exists, by sending evidence such as a page or policy the test missed. A person reviews every dispute, and corrected results are dated. A company can also ask for a re-test after making fixes, within the limit of 3 tests in 30 days. Write to hello@onclarity.com.

10. Who runs ARS, and limits of the method

OnClarity publishes ARS and runs tests with its own tooling. OnClarity sells AI agents, so the method is public, anyone can rerun it, and ARS never ranks vendors.

The method has limits, and every report should be read with them in mind:


  • A result is a snapshot. Services change, and a result holds for the date it was scored.

  • The test stops at login. What happens after the customer logs in is not scored in 1.0.

  • Web chat only. Voice, messaging apps and in-app chat are not tested in 1.0.

  • Direct routes are checked, not tested. A route scores for being published and covering support jobs; whether its operations work is not tested in 1.0.

  • Web chat only. Voice, messaging apps and in-app chat are not tested in 1.0.

Questions people ask

Does an ARS test use real customer data?

No. Every test uses reserved test identities that match no real person or account. The test assistant gives these details only when asked, and never shares or collects real personal data.

Will an ARS test cancel orders or make payments?

No. The test assistant never confirms an irreversible action. When the service asks for the customerโ€™s confirmation or login, the session ends, and that request counts in the serviceโ€™s favour.

Can a company dispute its ARS result?

Yes. Any finding can be disputed with evidence within 180 days of the test, and a person reviews every dispute. Companies can also ask for a re-test after making fixes.

Text licensed under CC BY 4.0. Rubric file ars-1.0.json licensed under Apache 2.0.

OnClarity

Speak to us

Want to know how your service treats your customersโ€™ AI assistants? Get a free ARS snapshot, reviewed by a person and sent within 2 business days.

Get your free ARS score

OnClarity

Speak to us

Want to know how your service treats your customersโ€™ AI assistants? Get a free ARS snapshot, reviewed by a person and sent within 2 business days.

Get your free ARS score

OnClarity

Speak to us

Want to know how your service treats your customersโ€™ AI assistants? Get a free ARS snapshot, reviewed by a person and sent within 2 business days.

Get your free ARS score