Do You Read Me? | A website-to-AI-bot experiment

Do You Read Me? is not presented as a controlled scientific study. It is a practice-based exploration of how websites can communicate with AI bots and whether identifiable bots request the machine-readable resources offered to them.

The project distinguishes three main functions. AI training crawlers collect content that may contribute to future versions of AI models. AI search and index crawlers collect and organise web content so that current AI systems can use it when answering questions. User-triggered AI retrieval bots visit websites in response to a specific user question or request, potentially retrieving the latest information directly. These are functional categories rather than fixed boundaries: a provider may use one agent for several purposes. The project also documents data-use controls such as Google-Extended and Applebot-Extended, but treats them as policy tokens rather than bots because they do not make requests themselves.

The field is changing rapidly. Technical conventions, crawler behaviour and provider policies may change while the project is running. Fixed conclusions may therefore have a limited lifespan. The project instead focuses on what can be observed, how it is measured, and what the results can and cannot tell us.

The first phase consists of creating and publishing the messengers and establishing the technical environment in which they can be observed. Writing these resources is itself part of the inquiry. It reveals ambiguities in specifications, differences between proposals and standards, and practical questions about locations, formats and crawler instructions. What is learned during implementation will inform the design of the monitoring, classification and analysis.

The project follows an iterative development process: build, observe, learn and adjust.

The project description is consequently a living document. It will be revised as observations produce new knowledge. To be continued...