Add /robots.txt with crawl + AI crawler rules

Author: edtechdev

robots.txt

@@ -0,0 +1,60 @@
+# robots.txt for edtechdev.github.io
+# The AI in Education wiki lives at /aied/ and is fully crawlable.
+# Policy: all crawlers allowed, including AI/LLM crawlers (the site is agent-ready).
+
+# Default: allow all
+User-agent: *
+Allow: /
+
+# --- AI / LLM crawlers: explicitly allowed (matches /aied/robots.txt policy) ---
+User-agent: GPTBot
+Allow: /
+
+User-agent: OAI-SearchBot
+Allow: /
+
+User-agent: ChatGPT-User
+Allow: /
+
+User-agent: ClaudeBot
+Allow: /
+
+User-agent: Claude-Web
+Allow: /
+
+User-agent: Google-Extended
+Allow: /
+
+User-agent: Googlebot
+Allow: /
+
+User-agent: Bingbot
+Allow: /
+
+User-agent: PerplexityBot
+Allow: /
+
+User-agent: Amazonbot
+Allow: /
+
+User-agent: Applebot
+Allow: /
+
+User-agent: Applebot-Extended
+Allow: /
+
+User-agent: cohere-ai
+Allow: /
+
+User-agent: CCBot
+Allow: /
+
+User-agent: Meta-ExternalAgent
+Allow: /
+
+# Site map
+Sitemap: https://edtechdev.github.io/aied/sitemap-index.xml
+
+# Signals that the site permits AI training, search, and AI input (agent-ready)
+Content-Signal: ai-train=yes, search=yes, ai-input=yes
+Schemamap: https://edtechdev.github.io/aied/schema/schemamap.xml