The concept of LLMS.txt files is still pretty new.
In fact, at the time of writing, it's merely a proposal – it's not a mandatory piece of information that's being recommended by LLMs.
In any case, with companies like Yoast now providing a one-step integration, now is a good time to understand the purpose of an LLMS.txt file and its potential (future) application.
What is an LLMS.txt File?
An LLMS.txt file is being touted as the LLM equivalent of a robots.txt file – but that's not how it works at all.
A robots.txt file is used to control how bots crawl and access a website. A robots.txt file essentially puts a perimeter around a website and informs bots and crawlers which areas they are allowed to access and which ones they should avoid.
The purpose of a LLMS.txt file is completely different to a robots.txt
An LLMS.txt file does not block or restrict crawler behaviour. An LLMS.txt file is simply a markdown text file that summarises key content on your website that is intended to help large language models (LLMs) to access and understand content on your website.
The theory is that an LLMS.txt file will make it easier for LLMs to understand and surface content on your website inside large language models' results.
LLMS.txt File Format: example.com/llms.txt
Do you need an LLMS.txt File to be cited in LLMs?
No, you do not need an LLMS.txt File to appear or be cited in LLMs like ChatGPT, Google Gemini, Claude, and Perplexity.
Without downplaying whatever future importance an LLMS.txt file may hold, there is nothing to support the notion that LLMs are actively using this new file format to form opinions or cite sources.
Further to that point, at the time of writing, there is no data, evidence, or case studies that show a positive correlation between an llms.txt file and greater prominence in LLMs.
Google Search Advocate John Mueller recently commented on a Reddit thread where one user posted that they had added an LLMS.txt file to their website and noticed no changes to bot behaviour when visiting their website.
"AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it). To me, it's comparable to the keywords meta tag – this is what a site-owner claims their site is about … (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)"
Mueller raises the most obvious point – if LLMs are scanning pages for information, why do we need an LLMS.txt file summary?
The keywords meta tag – which has not been used to rank website since 2009 – is an apt example/comparison. It's something that was used (and abused) to the point where Google no longer used it because it was so unreliable. LLMs are often criticised for accuracy of information, so we would expect that in the case that an LLMS.txt file is used as a guide, it will still be cross-referenced with the cited pages for accuracy.
Should I Create an LLMS.txt File?
I mean, you can if you want.
It's not going to harm you, per se, if you create one.
In cross-checking some of the biggest websites on the web, it is very hard to find sites that are currently using an LLMS.txt File. I tend to take this as a pretty good indication of whether or not it is warranted/useful.
With that said – watch this space.
If we see buy in from LLMs like ChatGPT, Google Gemini, Claude, and Perplexity promoting the use of LLMS.txt File, then this could change.
