# Crawling policy for listcheckup.com. # # This file is the whole policy. Cloudflare was prepending a managed block of its own that # disallowed nine AI crawlers, and that block is now switched off (bot_management # is_robots_txt_managed = false) so what is written here is what is served. If AI crawlers ever # appear to be blocked again without this file changing, that setting is where to look. User-agent: * Allow: / # --------------------------------------------------------------------------------------------- # AI crawlers: the ones that cite may read, the ones that only train may not. # # The distinction is worth stating because it is a deliberate position rather than an oversight. # These pages exist to be found and quoted — a reference page about bounce rates has no value # sitting unread — and for a product nobody has heard of, being named in an answer somebody asked # for is distribution we cannot otherwise buy. # # The one structural thing to understand before editing: the wildcard above allows everything, so # ANY crawler not named below is already permitted. The Allow entries are therefore documentation # rather than mechanism — they record a decision and survive someone tightening the wildcard # later. The Disallow list is the part that has to be complete, because a training crawler nobody # thought to name is a training crawler with full access. # # The uncomfortable part, recorded honestly: ClaudeBot and Google-Extended do not separate # grounding from training. Allowing them to cite also allows them to train. That trade was made # knowingly — this is marketing copy written to be read, not proprietary material — and it is why # the Content-Signal below still reserves training rights in the one form that is machine # readable. # --------------------------------------------------------------------------------------------- # Reserves training rights under Article 4 of EU Directive 2019/790, and states plainly that # search and reference use are welcome. User-agent: * Content-Signal: search=yes,ai-input=yes,ai-train=no,use=reference # --- Read, answer, attribute. Allowed. ------------------------------------------------------- # Anthropic. ClaudeBot is the crawler; the other two are search and user-initiated fetches. User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / # OpenAI, retrieval only. GPTBot — the training crawler — is disallowed further down. User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / # Google. Google-Extended governs Gemini grounding and training together; Googlebot itself is # ordinary search and is covered by the wildcard. User-agent: Google-Extended Allow: / User-agent: Google-CloudVertexBot Allow: / # Perplexity: the index, and user-initiated fetches, which are separate agents. User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / # Apple. Applebot plain is Siri and Spotlight search — that is citation, and it is a different # agent from Applebot-Extended, which governs Apple Intelligence training and is disallowed. User-agent: Applebot Allow: / # Amazon. Moved here from the blocked list after checking what it actually does: Amazon documents # Amazonbot as the crawler that lets Alexa answer questions, which is citation, not bulk # collection. It was disallowed on a wrong assumption and that was a misclassification. User-agent: Amazonbot Allow: / # Meta, user-initiated. Distinct from meta-externalagent, which is the training crawler and stays # disallowed. The names are confusingly close and are easy to conflate. User-agent: Meta-ExternalFetcher Allow: / # DuckDuckGo's assistant, and Mistral's user-initiated fetch. User-agent: DuckAssistBot Allow: / User-agent: MistralAI-User Allow: / # --- Bulk collection for training, nothing returned. Not allowed. ----------------------------- # # This is the list that matters, because anything omitted from it is allowed by the wildcard. # Add to it rather than assuming a new crawler is covered. User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: Bytespider Disallow: / User-agent: meta-externalagent Disallow: / User-agent: FacebookBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: Omgili Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / User-agent: Webzio-Extended Disallow: / User-agent: PanguBot Disallow: / User-agent: AI2Bot Disallow: / User-agent: AI2Bot-Dolma Disallow: / User-agent: Kangaroo Bot Disallow: / # --------------------------------------------------------------------------------------------- # Application surfaces carry no content worth indexing. The pages themselves also send # noindex, which is what actually removes them: a URL disallowed here can still be listed # without a description if something links to it, and the marketing pages link to sign-in # and register on purpose. Crawling is allowed so that instruction can be read. # --------------------------------------------------------------------------------------------- User-agent: * Allow: /app/account/login Allow: /app/account/register Disallow: /app/ Disallow: /v1/ Disallow: /swagger Sitemap: https://listcheckup.com/sitemap.xml