Rendered at 06:57:26 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
CqtGLRGcukpy 12 hours ago [-]
Most of what the author is doing won't actually help.
> LLMs ignore Markdown files — log analysis showed no evidence that GPTBot, ClaudeBot, or similar crawlers request or prioritise .md content, even when listed in llms.txt.
One trap when you name AI bots specifically is that is that robots.txt groups don’t merge. Crawlers will only follow the most specific group so if GPT bot has its own group named nothing under User-agent: * applies to it anymore so you need to repeat every disallowed line under each group.
Hissigh 1 days ago [-]
Are llms.txt actually read by bots now? I found one experiment that showed that no bot has accessed the file in 3 months.
I get the odd one try on my personal site, but usually its just robots.txt... /shrug
literallyroy 12 hours ago [-]
I once had oh-my-pi read a website’s llms.txt while doing research many months ago.
Muhammad523 15 hours ago [-]
why? You're helping AI companies to take your writing for free
mmcallister 12 hours ago [-]
Whilst I agree with most criticism of AI/LLMs but most of the time at the other end of someone using Claude/ChatGPT there is still a human there, a human that is trying to solve a problem or learn something. Maybe I'm naive and it's just all bots scraping sites to build their corpus of data, but at least the way I use AI is to solve problems and build things I couldn't do before (either because of time, or it was out of my reach)
> LLMs ignore Markdown files — log analysis showed no evidence that GPTBot, ClaudeBot, or similar crawlers request or prioritise .md content, even when listed in llms.txt.
https://www.longato.ch/llm-md-files/
> Across all 50 sites, across 200+ days, across every major AI user-agent logged - the `/llms.txt` endpoint was requested zero times.
https://mishamanko.com/research/what-ai-bots-actually-read
https://www.rebootonline.com/geo/llms-txt-experiment/