Files
worldmonitor/public/robots.txt
Elie Habib 0169245f45 feat(seo): BlogPosting schema, FAQPage JSON-LD, extensible author system (#2284)
* feat(seo): BlogPosting schema, FAQPage JSON-LD, author system, AI crawler welcome

Blog structured data:
- Change @type Article to BlogPosting for all blog posts
- Author: Organization to Person with extensible default (Elie Habib)
- Add per-post author/authorUrl/authorBio/modifiedDate frontmatter fields
- Auto-extract FAQPage JSON-LD from FAQ sections in all 17 posts
- Show Updated date when modifiedDate differs from pubDate
- Add author bio section with GitHub avatar and fallback

Main app:
- Add commodity variant to middleware VARIANT_HOST_MAP and VARIANT_OG
- Add commodity.worldmonitor.app to sitemap.xml
- Shorten index.html meta description to 136 chars (was 161)
- Remove worksFor block from index.html author JSON-LD
- Welcome all bots in robots.txt (removed per-bot blocks, global allows)
- Update llms.txt: five variants listed, all 17 blog post URLs added

* fix(seo): scope FAQ regex to section boundary, use author-aware avatar

- extractFaqLd now slices only to the next ## heading (was: to end of body)
  preventing bold text in post-FAQ sections from being mistakenly extracted
- Avatar src now derived from DEFAULT_AUTHOR_GITHUB constant (koala73)
  only when using the default author; custom authors fall back to favicon
  so multi-author posts show a correct image instead of the wrong profile
2026-03-26 12:48:56 +04:00

11 lines
259 B
Plaintext

# World Monitor - protect API routes from crawlers
User-agent: *
Allow: /
Allow: /api/story
Allow: /api/og-story
Disallow: /api/
Disallow: /tests/
Sitemap: https://www.worldmonitor.app/sitemap.xml
Sitemap: https://www.worldmonitor.app/blog/sitemap-index.xml