robots.txt
Definition
A plain text file at the root of a website that tells crawlers, including AI crawlers, which parts of the site they may or may not access.
Why it matters
Blocking an AI crawler in robots.txt can remove a brand's own pages from that system's source material entirely, which is sometimes intentional and sometimes an accidental side effect of a blanket bot-blocking rule.
Example
A site that blocks GPTBot in robots.txt will not have its pages crawled for that purpose, even if the rest of the site is fully public.
Related terms
Sources
AI crawler
An AI crawler is an automated program, run by an AI company, that fetches web pages to train models, build a search index or answer a user's request.
Optimization
llms.txt
llms.txt is a proposed convention for a plain text file at a site's root that summarises the site and links to its key pages for language models.
Keep learning
What is robots.txt?
A plain text file at the root of a website that tells crawlers, including AI crawlers, which parts of the site they may or may not access.
Why does robots.txt matter?
Blocking an AI crawler in robots.txt can remove a brand's own pages from that system's source material entirely, which is sometimes intentional and sometimes an accidental side effect of a blanket bot-blocking rule.
See these terms measured on your brand.
Start with a first audit. We show where you appear, who is named instead, and what to fix first.