mirror of
https://github.com/Patiun/com.gregkilmer.www.git
synced 2026-11-08 04:08:24 +00:00
Claude
a2f723ca2d
Phase 1 of the functional design plan. - Move the old site into legacy/, unchanged, for reference. It still builds from there until cutover. - Add packages/schema: the one definition of all 20 note types, with field titles and descriptions, note and vault checks, and JSON Schema export so later editing tools and the MCP server share it. - Add vault-kit: Obsidian templates generated from the schema, and the Site guide for 24.01_PersonalWebsite. - Add tools/migrate: turns the legacy JSON, collections, store data and page text into 545 Site folder notes and 84 images, and checks the result with the schema. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012aSbo2D1qP8cMFQCykf7kx
95 lines
2.1 KiB
Plaintext
95 lines
2.1 KiB
Plaintext
# =============================================================================
|
|
# robots.txt — gregkilmer.com
|
|
# =============================================================================
|
|
# Strict anti-scraping policy.
|
|
#
|
|
# - Google/Bing/etc. (legitimate search) are ALLOWED for SEO.
|
|
# - /api/ and /contact are disallowed for everyone (the form + endpoints
|
|
# should not be indexed or crawled).
|
|
# - Known AI training bots are blocked by User-Agent below. Note that
|
|
# robots.txt is voluntary; determined scrapers will ignore it. This is a
|
|
# first layer (paired with noindex on /contact and email obfuscation).
|
|
# =============================================================================
|
|
|
|
# --- Default policy: allow crawling of public pages, block the API + contact
|
|
User-agent: *
|
|
Allow: /
|
|
Disallow: /api/
|
|
Disallow: /contact
|
|
|
|
# --- Block known AI training / scraping bots explicitly ---------------------
|
|
# OpenAI
|
|
User-agent: GPTBot
|
|
Disallow: /
|
|
|
|
User-agent: ChatGPT-User
|
|
Disallow: /
|
|
|
|
User-agent: OAI-SearchBot
|
|
Disallow: /
|
|
|
|
# Anthropic
|
|
User-agent: ClaudeBot
|
|
Disallow: /
|
|
|
|
User-agent: Claude-Web
|
|
Disallow: /
|
|
|
|
User-agent: anthropic-ai
|
|
Disallow: /
|
|
|
|
# Google's AI/Vertex crawlers (separate from regular Googlebot, which is allowed)
|
|
User-agent: Google-Extended
|
|
Disallow: /
|
|
|
|
# Common Crawl (feeds many AI training datasets)
|
|
User-agent: CCBot
|
|
Disallow: /
|
|
|
|
# ByteDance / TikTok
|
|
User-agent: Bytespider
|
|
Disallow: /
|
|
|
|
# Meta
|
|
User-agent: meta-externalagent
|
|
Disallow: /
|
|
|
|
User-agent: Meta-ExternalAgent
|
|
Disallow: /
|
|
|
|
User-agent: FacebookBot
|
|
Disallow: /
|
|
|
|
# Amazon / Alexa
|
|
User-agent: Amazonbot
|
|
Disallow: /
|
|
|
|
# Apple's AI/Applebot-Extended (regular Applebot remains allowed for Siri/Suggestions)
|
|
User-agent: Applebot-Extended
|
|
Disallow: /
|
|
|
|
# Perplexity
|
|
User-agent: PerplexityBot
|
|
Disallow: /
|
|
|
|
# Cohere
|
|
User-agent: cohere-ai
|
|
Disallow: /
|
|
|
|
# Diffbot
|
|
User-agent: Diffbot
|
|
Disallow: /
|
|
|
|
# Imagesift / OC-72 / other image scrapers
|
|
User-agent: ImagesiftBot
|
|
Disallow: /
|
|
|
|
User-agent: omgili
|
|
Disallow: /
|
|
|
|
User-agent: Omgilibot
|
|
Disallow: /
|
|
|
|
# Sitemap (uncomment once a sitemap.xml is generated)
|
|
# Sitemap: https://gregkilmer.com/sitemap.xml
|