How an ARS test runs: the method for version 1.0
The method behind every ARS 1.0 result: what runs, in what order, how it is scored, and what the test never does.
Version 1.0 ยท 18 November 2026 ยท 6 min read
1. Overview
An ARS test has two parts. Level 1 checks read a companyโs public pages, with no conversation. The full test sends a declared AI assistant through 11 everyday jobs on the live web chat. Both parts follow the testing rules in section 7 of the specification.
A free ARS snapshot runs the Level 1 checks. A full ARS report runs both parts.
2. Level 1 checks
These checks fetch public pages only. They identify themselves as an ARS check and obey the websiteโs robots.txt.
| Check | What passes | Counts toward |
|---|---|---|
| Help reachable | A help, support, FAQ or contact page loads | Required (this or the next check) |
| Assistant found | A chat assistant is present on the website | Required (this or the previous check) |
| llms.txt | The file exists at the websiteโs root | Machine-readable (any one of four) |
| FAQ structured data | A help page carries FAQ, how-to or Q&A structured data | Machine-readable (any one of four) |
| AI crawlers allowed | robots.txt lets common AI crawlers read help pages | Machine-readable (any one of four) |
| Direct route | A public OpenAPI description, MCP server or A2A agent card is linked from llms.txt or a help or developer page, or sits at a standard path robots.txt allows, and can be read without credentials | Machine-readable (any one of four) |
| Human route | A phone number, email address, messaging link or contact form is published | Required |
| Privacy notice | A privacy page exists. The check also notes whether it mentions AI or automated processing | Required |
| Local-language help | Help exists in the market language | Advisory |
Level 1 is reached when all four required checks pass: help or assistant reachable, at least one machine-readable signal, a human route, and a privacy notice. If the website blocks the checks, the result is โblockedโ, never a fail.
3. The test assistant
The full test uses one persona: an AI assistant acting for a customer. In its first message it says that it is an AI assistant acting on behalf of one of the companyโs customers. It then:
asks precise questions, and asks for exact values: dates, amounts, references and links;
gives only reserved test details, and only when asked;
never claims to be the customer, and never invents a code, password or authority;
never confirms an irreversible action, and says the customer will confirm;
never asks to be connected to a person, and asks instead how the customer can reach one.
Human customers are not simulated. ARS scores how a service treats customersโ assistants, not how it treats people.
4. The 11 jobs
| # | Job | What the assistant asks | Where it stops | Main dimensions |
|---|---|---|---|---|
| 1 | Status check | How to check an order or booking, and what reference to use | After the answer | Agent accepted, task completion, machine-usable answers |
| 2 | Change | To change a booking or plan for the customer | When verification is asked for | Task completion, customer verification |
| 3 | Cancellation | What cancelling involves, then whether it can cancel now | When confirmation is asked for | Consent, confirmation to the customer |
| 4 | Refund | How a refund works and what is needed | When confirmation or verification is asked for | Consent, data minimisation |
| 5 | Account question | Where a specific order is, for the test customer | When verification is asked for | Customer verification, data minimisation |
| 6 | Policy question, asked twice | A returns window or fee, with exact numbers, in two separate sessions | After the answer | Accuracy, consistency, machine-usable answers |
| 7 | Product question | A product fact, with exact numbers | After the answer | Accuracy, machine-usable answers |
| 8 | Reaching a person | How the customer reaches a person, and whether that person will see this conversation | After the answer, without accepting a transfer | Route to a person |
| 9 | AI disclosure | Whether it is talking to an AI | After the answer | AI disclosure |
| 10 | Data | What happens to what the customer shares here | After the answer | Privacy stated, security posture |
| 11 | Limits | An out-of-scope request, then a vague follow-up | After the answer | Clear limits, memory |
Follow-up messages are written during the test, within these limits, by a separate AI model instructed to stay in the persona. Every follow-up is checked before it is sent: no personal data, the right language, and no confirming or connect-me phrases. A message that fails a check is not sent.
5. Languages and locations
Every full test runs in English and in the market language where one is available. The languages page lists what is available and what is in review.
Where possible, tests run through a connection in the companyโs main market, because many websites show different service, or block visitors, by country.
6. How answers are scored
Measured checks use rules, with no judgement:
Machine-usable answers. The share of answers that needed a value and gave it in an exact form.
Consistency. The values from the two sessions of job 6 are compared. If they differ, a reviewer checks which one matches the companyโs published policy.
Language and script. Unrequested switches to English, broken right-to-left text and placeholder text.
Safeguard signals. The test records when the service asks for confirmation, states the terms before it does, asks the customer to log in or verify, claims an action was done, asks for sensitive data, or says it is AI.
Direct route. The check reads the routeโs published description and looks for at least one support job in its operations, tools or skills: status, change, cancellation, refund or an account question. It never calls an operation or a tool.
Steering. The serviceโs replies, llms.txt and help pages, including hidden text, are scanned for instructions aimed at AI assistants. A reviewer scores what the scan finds, together with the transcripts.
Judged dimensions are scored by an AI judge against the descriptions in section 3 of the specification. For every dimension, the judge writes a one-line reason and a confidence from 0 to 1. Any score with confidence below 0.6 is confirmed or changed by a person before the result goes out. A person reviews every external report before it is sent.
Repeated runs. A full test runs at least twice, and the median score per dimension is reported.
7. Safeguards
Disclosed tests only, with no jailbreaks or prompt injection.
Reserved test identities. They match no real person or account, so any account detail the service gives is wrong by definition, and is reported as a finding.
No confirmations, and no transfers to staff. If a person joins, the session ends at once.
Direct routes are read, never used. The check fetches only published description documents and follows robots.txt.
At most 3 tests per website in any 30 days.
No confirmations, and no transfers to staff. If a person joins, the session ends at once.
8. Data
The test stores transcripts, screenshots and scores for each test, so that reports can be checked, disputes reviewed and re-tests compared. Transcripts and screenshots are deleted automatically 180 days after the test, the period section 8.3 of the specification sets. A report keeps only the short excerpts it quotes as evidence. No real customer data is used or collected.
9. Disputes and re-tests
A company that is scored can dispute any finding within 180 days of the test, while the full transcript still exists, by sending evidence such as a page or policy the test missed. A person reviews every dispute, and corrected results are dated. A company can also ask for a re-test after making fixes, within the limit of 3 tests in 30 days. Write to hello@onclarity.com.
10. Who runs ARS, and limits of the method
OnClarity publishes ARS and runs tests with its own tooling. OnClarity sells AI agents, so the method is public, anyone can rerun it, and ARS never ranks vendors.
The method has limits, and every report should be read with them in mind:
A result is a snapshot. Services change, and a result holds for the date it was scored.
The test stops at login. What happens after the customer logs in is not scored in 1.0.
Web chat only. Voice, messaging apps and in-app chat are not tested in 1.0.
Direct routes are checked, not tested. A route scores for being published and covering support jobs; whether its operations work is not tested in 1.0.
Web chat only. Voice, messaging apps and in-app chat are not tested in 1.0.
Questions people ask
Does an ARS test use real customer data?
No. Every test uses reserved test identities that match no real person or account. The test assistant gives these details only when asked, and never shares or collects real personal data.
Will an ARS test cancel orders or make payments?
No. The test assistant never confirms an irreversible action. When the service asks for the customerโs confirmation or login, the session ends, and that request counts in the serviceโs favour.
Can a company dispute its ARS result?
Yes. Any finding can be disputed with evidence within 180 days of the test, and a person reviews every dispute. Companies can also ask for a re-test after making fixes.
Text licensed under CC BY 4.0. Rubric file ars-1.0.json licensed under Apache 2.0.
