AI crawler
Also known as: GPTBot, AI bot, AI user agent
Definition
An AI crawler is an automated program, run by an AI company, that fetches web pages to train models, build a search index or answer a user's request.
Why it matters
Vendors publish separate crawlers for separate purposes. Blocking a search crawler can remove a site from that assistant's search answers, while blocking a training crawler is a different decision.
Example
A site allows a vendor's search crawler but disallows its training crawler in robots.txt.
Related terms
Optimization
llms.txt
llms.txt is a proposed convention for a plain text file at a site's root that summarises the site and links to its key pages for language models.
Concepts
Grounding
Grounding is connecting an AI model's answer to external sources, such as search results, so that claims can be supported and cited.
Optimization
Structured data
Structured data is markup, commonly schema.org in JSON-LD, that describes the meaning of a page's content in a form machines can read.
What is AI crawler?
An AI crawler is an automated program, run by an AI company, that fetches web pages to train models, build a search index or answer a user's request.
Why does ai crawler matter?
Vendors publish separate crawlers for separate purposes. Blocking a search crawler can remove a site from that assistant's search answers, while blocking a training crawler is a different decision.
See these terms measured on your brand.
Start with a first audit. We show where you appear, who is named instead, and what to fix first.