# Korely robots.txt
#
# Strategy: allow everything, to everyone, everywhere.
# Rationale: Korely is unknown. The scarce resource is not our content, it is
# people knowing we exist. Being in a model's training corpus is how it
# recommends us without browsing; being readable by the live-fetch bots is how
# it recommends us while browsing. We want both. We have no content to
# monetize and nothing to withhold.
#
# Evidence this matters: our first inbound user arrived saying "just got
# recommended Korely from ChatGPT" (2026-09). That path runs through the bots
# listed below.
#
# WARNING: keep Cloudflare's "Managed robots.txt" toggle OFF
# (dashboard > AI Crawl Control). When on, Cloudflare prepends its own block
# with `Disallow: /` for GPTBot, ClaudeBot, Google-Extended, CCBot and others,
# which contradicts every line of this file. Two opposite instructions for the
# same user-agent resolve differently crawler by crawler, so the file stops
# meaning anything. If AI training is ever to be restricted, edit THIS file,
# do not let a toggle write a second opinion above it.
#
# Blocking is enforced separately, in AI Crawl Control > Crawlers. robots.txt
# is a request; that tab is the bouncer. Both must stay open.

# === Search engines ===
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Slurp
Allow: /

User-agent: YandexBot
Allow: /

User-agent: Baiduspider
Allow: /

User-agent: PetalBot
Allow: /

# === Apple ===
User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

# === OpenAI ===
# GPTBot trains, OAI-SearchBot indexes, ChatGPT-User fetches live when a
# person asks a question. The third one is a human at the other end.
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# === Anthropic (3-bot split since 2024-2025) ===
User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: anthropic-ai
Allow: /

# === Perplexity ===
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# === Google AI ===
User-agent: Google-Extended
Allow: /

# === Microsoft Copilot ===
User-agent: msnbot
Allow: /

# === DuckDuckGo AI ===
User-agent: DuckAssistBot
Allow: /

# === Mistral ===
User-agent: MistralAI-User
Allow: /

# === Meta ===
User-agent: Meta-ExternalAgent
Allow: /

User-agent: Meta-ExternalFetcher
Allow: /

User-agent: FacebookBot
Allow: /

# === Other AI / LLM / agent crawlers ===
User-agent: Bytespider
Allow: /

User-agent: CCBot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: Diffbot
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: YouBot
Allow: /

User-agent: FirecrawlAgent
Allow: /

User-agent: ImagesiftBot
Allow: /

User-agent: Timpibot
Allow: /

# === Default: allow all ===
# Content Signals Policy (contentsignals.org) stated affirmatively: silence is
# read as "neither granted nor restricted", and we are granting.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# Sitemap (Astro generates an index that points to sitemap-0.xml)
Sitemap: https://korely.ai/sitemap-index.xml

# Agent-facing entry point (https://llmstxt.org): what Korely is, and how an
# agent signs itself up without a human in the loop.
# https://korely.ai/llms.txt