Do AI Bots Read What Websites Tell Them?

People think differently about AI, partly because they encounter it in different forms and settings. AI does not operate in isolation: its behaviour is shaped by the instructions, context and limits around it. A general-purpose tool used without clear guidance is more likely to disappoint or behave unpredictably than one configured for a specific task.

Yet many people feel they have little influence over how AI tools behave, whether in an individual exchange or during model training. When people and organisations do try to shape AI behaviour, they usually do so within the tool itself: through prompts and custom instructions, or through techniques such as system prompts, retrieval-augmented generation and guardrails.

This project explores a different possible point of influence: the website itself. Can a website be configured to help AI systems recognise and use its information more accurately, both when content is collected for possible training and when a bot visits in response to a user’s question?

Most AI providers benefit when their systems can accurately understand the text, images and structure of websites. Yet reading HTML alone does not always provide a clear or complete account of what a website means. Information may be fragmented, technically difficult to access or presented without enough structure to distinguish authoritative information from surrounding content.

AI bots encounter websites in different ways. Training crawlers collect content that may contribute to future models, while real-time retrieval bots visit websites in response to a current user question. This distinction matters, especially for websites that serve the public interest. AI can only help people with applications, complaints or objections when it finds and correctly represents the relevant procedures, conditions and limitations.

Websites can supplement their ordinary pages with machine-readable resources and structured markup. These signals may help automated systems discover content, understand its structure or determine how they are permitted to access it. This site calls them the messengers.

The messengers examined here are robots.txt, which communicates crawling rules; llms.txt and ai.txt, which explore emerging ways of addressing AI systems; security.txt, an established standard included as a point of comparison; and Schema.org, which adds structured data to ordinary web pages. Each messenger has its own page explaining its purpose, status and limitations.

The underlying question is whether AI bots make use of the information offered through these messengers. Do You Read Me? starts with the first observable step: do identifiable AI training crawlers and real-time retrieval bots request the URL-based messengers, and if so, how often?

A request shows that a bot accessed a resource. It does not prove that the information was understood, retained or used. Schema.org requires a different form of testing because its structured data is retrieved as part of the web page itself.

More about the research question, method and limitations can be found on the experiment page.