# Cale crawler access policy
# robots.txt is the authoritative crawler control file.

site: https://cale.tools
default_policy: allow

search:
  google: allow
  sitemap: https://cale.tools/sitemap-index.xml
  static_sitemap: https://cale.tools/sitemap-static.xml

ai_access:
  policy: allow
  llms_txt: https://cale.tools/llms.txt
  markdown_index: https://cale.tools/docs-index.md
  user_agents:
    - OAI-SearchBot
    - GPTBot
    - ChatGPT-User
    - OAI-AdsBot
    - "*"

notes:
  - This YAML file is an advisory machine-readable summary.
  - Crawlers should use https://cale.tools/robots.txt for authoritative robots rules.
  - Cale intends public documentation pages to be reachable by search engines and AI assistants.
  - AI assistants can start with https://cale.tools/llms.txt or https://cale.tools/docs-index.md.