# CarGeek
#
# Reading the reference needs no account and everything public is open to
# crawlers. What follows is not there to hide anything — it keeps crawl budget
# on the fifteen hundred pages worth indexing rather than on screens that are
# either private, personal, or a view of a page already listed.

User-agent: *

# Search, and AI answers that cite and link to the page they used, are welcome;
# training a model on the reference is not (the terms of use, section 6, say the
# same). Cloudflare's content-signals vocabulary, read by the crawlers that
# honour it and ignored as an unknown line by the ones that do not.
Content-Signal: search=yes, ai-input=yes, ai-train=no

# Behind authentication. A crawler only ever gets a redirect to the sign-in
# form from these, and every one of those is a wasted request.
Disallow: /settings/
Disallow: /moderation/
Disallow: /admin/
Disallow: /notifications
Disallow: /feed
Disallow: /saved

# Auth screens. Nothing to index, and /register in particular tends to be
# crawled far more often than it deserves.
Disallow: /login
Disallow: /register
Disallow: /forgot-password
Disallow: /reset-password
Disallow: /verify-email
Disallow: /user/

# Livewire's own endpoint. Not a page.
Disallow: /livewire/

# The offline fallback exists for the service worker, not for readers.
Disallow: /offline

# Filtered and searched views of a listing, which are the same pages in a
# different order. Every one of them already carries a canonical pointing at
# the unfiltered page, so none was ever going to be indexed — but a canonical
# is only read *after* the fetch, and the fetch is the thing being paid for.
#
# Measured 2026-09-04: Google's crawl of this site ran 77% "discovery" — new
# URLs rather than refreshes of known ones — while 8,035 discovered URLs sat
# unindexed and the crawl rate for the whole domain fell 96% in eight days.
# `/{locale}/codes?severity=moderate` and `/{locale}/diagnose?q=<phrase>` are
# two of the shapes that inflate that surface: the symptom directory alone
# links 199 query URLs, and there are ten locale trees.
#
# **These four are the parameters Googlebot has actually fetched**, counted
# from the origin log rather than guessed from the code: `q` 201, `subsystem`
# 125, `severity` 37, `emissions` 4. The first guess at this list named `page`,
# `sort`, `kind` and `system` — none of which the site emits — and would have
# blocked nothing while reading as a fix.
#
# **`id` is deliberately absent and must stay absent.** It is the third most
# crawled parameter here (65 fetches) and it is not a facet at all: it is the
# cache-busting version on `livewire.min.js` and the built assets. Blocking it
# would stop Googlebot fetching the JavaScript the pages need to render.
#
# The sitemap contains no query strings at all, so nothing listed here removes
# a page anybody asked Google to index.
Disallow: /*?q=
Disallow: /*&q=
Disallow: /*?subsystem=
Disallow: /*&subsystem=
Disallow: /*?severity=
Disallow: /*&severity=
Disallow: /*?emissions=
Disallow: /*&emissions=

Allow: /

# Backlink and dataset crawlers, blocked on evidence rather than principle.
#
# Measured over one day of access logs on 2026-08-09: 11,178 requests reached
# the origin and **7,452 of them — 67% — were these four**. None sends a reader
# here. MJ12bot alone was 4,033, more than every human, every search engine and
# every locale tree put together; Amzn-SearchBot was 2,328.
#
# Amazon is the instructive one. The training block below lists `Amazonbot`,
# and this crawler identifies as **`Amzn-SearchBot`** — a different token, so
# that rule never matched it. A blocked-looking bot that is not blocked is worse
# than an unblocked one, because nobody re-checks.
#
# Search engines are deliberately untouched: Googlebot, bingbot and Yandex are
# how the corpus is found, and their 1,000-odd daily requests are the point of
# the sitemap.
#
# **This is a request, not a wall.** All four claim to honour robots.txt; if the
# numbers do not fall within a few days, the enforcement layer is a Cloudflare
# WAF rule, which is in the dashboard rather than in this repo. Re-measure
# before assuming it worked.

User-agent: MJ12bot
Disallow: /

User-agent: Amzn-SearchBot
Disallow: /

User-agent: SERankingBacklinksBot
Disallow: /

User-agent: serpstatbot
Disallow: /

# AI training crawlers, refused by name — the posture settled on 2026-08-25:
# allow every crawler at the edge, refuse training here. **Until 2026-10-20 this
# list lived in Cloudflare's managed robots.txt, prepended at the edge, and that
# block had silently stopped being served**: the live file refused no AI
# crawler at all while this file's own comment above still said it did. So the
# list lives in the repo now, where a test can read it. Search and answer
# crawlers — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot — are
# deliberately absent: ChatGPT is the site's largest source of readers.
# `Google-Extended` is a robots.txt token with no crawler of its own, so
# refusing it costs Search nothing.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

Sitemap: https://cargeek.org/sitemap.xml