Agent readability checker
Check if AI agents and crawlers can find and read a page.
Questions
What is llms.txt?
A Markdown file at /llms.txt that lists the important pages of a site with a short summary of each. It was proposed at llmstxt.org. Agents read it to find the right page without crawling the whole site.
How do I block or allow an AI crawler?
Add a User-agent group for the crawler in robots.txt, with Disallow: / to block it or Allow: / to allow it. Crawlers that collect training data, like GPTBot and ClaudeBot, are separate from the ones that fetch a page when a person asks, like ChatGPT-User and Claude-User, so you can block one and allow the other.
Why serve a Markdown version of a page?
An HTML page carries navigation, scripts and styles, and an agent pays for all of it in tokens. A Markdown copy has only the content. Serve it at the same path with .md at the end, link it with <link rel="alternate" type="text/markdown">, or return it when the request has Accept: text/markdown.
Do AI crawlers run JavaScript?
Many do not. If scripts add the text of a page, those crawlers see an empty page. Render the text on the server or at build time.
Are the links I check saved?
No. The page and its files are fetched when you press Check, and nothing is stored.
How it works
The page at the link is fetched from our server without running its scripts, the way crawlers that do not run JavaScript see it. Then robots.txt, llms.txt, llms-full.txt, the sitemap and the Markdown version of the page are fetched from the same site.
The crawler list reads the robots.txt rules for each crawler and the path of the page. Some sites also block crawlers at the firewall, which this check cannot see.
Company of Craft also builds 
stateofpixel, which catches UI regressions before they merge.