Authored by Michael Shaskey of Klove.ai
Published May 16th 2026
Michael Shaskey
Three independent studies, covering more than 300,000 domains between them, have now looked at whether having a llms.txt file improves your citation rate in ChatGPT, Perplexity, or Google AI Overviews. All three reached the same conclusion: it doesn't. The file that was supposed to be the robots.txt of the generative era turns out to be closer to the meta keywords tag — a declaration of intent that the systems it addresses have quietly decided to ignore.
The adoption story was convincing enough to build an ecosystem around it. Anthropic, Cloudflare, Stripe, and Vercel all published llms.txt files. Tooling vendors built systems to support the format. Press outlets ran how-to guides. However, the empirical record, once you read past the advocacy, is flat.
This essay covers what the data shows, why the file cannot do what its proponents claim, the narrower case where it does have genuine value, and the four things that move the citation needle instead.
Three studies covering 300,000+ domains found zero statistically significant correlation between llms.txt adoption and AI citation frequency.
No major AI provider — not OpenAI, Anthropic, or Google — has documented llms.txt as an input to their citation pipeline.
The file does have genuine value for agentic pipelines: it lowers integration friction and reduces token cost in RAG systems.
According to the Princeton/Georgia Tech ACM KDD 2024 study, citing external sources improved AI visibility by 115% for lower-ranked content — the single largest lever tested.
Promotional language reduces AI citation rates by 26%, per Semrush data. Content that gets cited reads like reference material, not marketing copy.
llms.txt refers to a simple Markdown file placed at a domain's root — a curated list of key URLs with plain-prose descriptions, designed to help large language models understand a site's content without parsing cluttered HTML. Jeremy Howard proposed the format in 2024, and the robots.txt analogy was immediate: familiar format, minimal cost, repurposed for the generative era.
Credibility arrived through association. When Anthropic, Cloudflare, Stripe, and Vercel adopted the format, smaller teams reasonably concluded that major AI providers must be reading it. By July 2025, roughly 950 domains had implemented the file, giving the standard visible momentum. Across the 300,000 domains SE Ranking measured, adoption sat at around 10.13% — spread evenly across traffic tiers, which matters because adoption patterns cannot explain citation outcomes.
Yet nobody verified whether any major AI provider had actually documented the file as an input to their crawler or citation pipeline. None had — not OpenAI, not Anthropic, not Google. In other words, the credibility came from the brand names of early adopters, not from confirmed platform support. That gap between adoption momentum and confirmed mechanism is where the empirical work becomes damaging.
The data says no. Three independent studies found no measurable link between llms.txt adoption and AI citation frequency in ChatGPT, Perplexity, or Google AI Overviews.
The most rigorous analysis to date comes from SE Ranking's 300,000-domain study. The researchers used XGBoost machine learning alongside traditional statistical methods to test whether llms.txt predicted AI citation frequency. The result was striking: removing the llms.txt variable from the model improved its accuracy. When a feature's absence makes your model more accurate, the feature is adding noise, not signal.
OtterlyAI ran a parallel experiment over 90 days, monitoring 62,100 individual AI-bot visits. They tracked whether bots that fetched llms.txt produced different citation behaviour from those that didn't. Their conclusion: the file's impact is "marginal at best." As a result, they removed the LLMs checker from their GEO Audit product entirely.
The third data point comes from a longitudinal study documented by Search Engine Land, tracking 10 sites for 90 days before and after llms.txt implementation. Eight of nine sites showed no measurable change. The one apparent outlier had run a concurrent PR campaign and technical SEO fixes during the same window — confounders that make attributing any lift to the file impossible.
Summary of llms.txt citation-impact studies | |||
|---|---|---|---|
Study | Scale | Duration | Finding |
SE Ranking | 300,000 domains | Cross-sectional | Zero correlation; removing the variable improved model accuracy |
OtterlyAI | 62,100 AI-bot visits | 90 days | "Marginal at best"; removed from their GEO Audit |
Search Engine Land | 10 sites | 90 days pre/post | 8 of 9 sites showed no measurable citation change |
On the platform side, Google's Gary Illyes has stated on record that the company has "no plans to support LLMs.txt". Additionally, OpenAI's crawler documentation focuses exclusively on robots.txt as the relevant access-control signal. The file is a proposed standard, not an adopted one.
SE Ranking's 300,000-domain analysis found no statistically significant difference in AI citation frequency between domains with and without llms.txt. In contrast, the Princeton/Georgia Tech ACM KDD 2024 study found adding statistics lifted visibility by 41%.
The reason is structural, not accidental. AI citation behaviour in systems like ChatGPT or Perplexity comes primarily from training data — the web as it existed at the training cutoff, weighted by signals the model learned to associate with trustworthiness. A file placed at a domain root in 2024 cannot retroactively change what a model absorbed during training in 2023.
Furthermore, even when GPTBot fetches a llms.txt file, there is no documented pipeline connecting "file fetched" to "content cited in response." Fetching and citing are separate systems. The crawler reads; the language model responds based on patterns from training. A declarative Markdown file cannot bridge those two processes.
Google's AI Overviews and AI Mode compound this problem. Google has not indicated that llms.txt is used as a signal in AI Overviews or AI Mode. Both rely on traditional search signals: E-E-A-T (experience, expertise, authoritativeness, trustworthiness), backlink authority, content structure, and query relevance. Google's John Mueller has drawn the comparison to the deprecated meta keywords tag: a file that declares intent loudly and carries no weight quietly.
The meta keywords analogy is more precise than it might appear. Meta keywords failed not because the idea was stupid — describing your content in structured metadata is reasonable — but because no search engine ever committed to treating that description as authoritative. llms.txt is in the same position: a reasonable idea waiting for platform adoption that has not arrived and may not.— Editor's note
AuthorityTech's synthesis of the empirical literature places llms.txt in what they call the "identity tier" — a set of signals that describe a site's self-reported identity. Across every study, that tier does not drive citations at scale. Instead, what drives citations is content that AI systems have independently learned to trust: structured, evidenced, specific prose that answers questions directly.
llms.txt provides genuine value in agentic pipelines — automated systems where AI agents browse, retrieve, and process web content on behalf of users. This is a different system from AI search citations, and confusing the two is where most guidance goes wrong.
OtterlyAI, despite their null finding on citation behaviour, identifies the genuine value precisely: lower integration friction. Partners and developer tools can ingest your content without writing custom scrapers. A well-structured llms.txt reduces token cost in RAG pipelines — RAG (retrieval-augmented generation) refers to systems that retrieve external documents and feed them into an LLM at query time — by providing curated Markdown rather than forcing parsers to strip boilerplate HTML. For a software company whose docs are consumed by other tools, this is a real operational benefit.
The tooling ecosystem reflects this use case. Mintlify and GitBook generate llms.txt automatically as part of their documentation infrastructure. If your CMS already produces the file, the marginal cost is zero and the agentic-layer benefit is real.
Apollo.io's /llm-info page illustrates a related pattern: a clean, crawlable entity summary that reduces misrepresentation in AI-generated answers about the company. That page is designed for the agentic browsing layer — autonomous agents doing research on behalf of users — not the training pipeline that determines what a model cites.
Therefore, the honest framing is: if your CMS generates the file automatically, leave it on. If implementing it requires manual effort, spend that time on content structure instead — because that's where the citation evidence points.
The most credible evidence comes from the Aggarwal et al. paper published at ACM KDD 2024 by researchers from Princeton, Georgia Tech, and IIT Delhi. The findings are specific enough to act on.
According to the Princeton/Georgia Tech research, three techniques produce the largest visibility lifts in generative engine results:
Citing external sources improved visibility by 115% for lower-ranked content — the biggest single lever in the study.
Adding statistics improved AI visibility by 41%.
Adding quotations improved it by 28%.
In contrast, keyword stuffing performed 10% worse than the baseline. AI systems actively penalise the technique that dominated SEO for a decade.
Citing authoritative external sources improved AI search visibility by 115% for lower-ranked content — the single largest lever identified across all optimisation techniques tested.— Aggarwal et al., ACM KDD 2024 (Princeton / Georgia Tech / IIT Delhi)
Content structure compounds these gains. Ahrefs data shows the share of AI Overview citations drawn from Google's top-10 results dropped from 76% in July 2025 to 38% by early 2026. As AI systems get better at assessing content quality independently of rank, structural clarity matters more and positional authority matters less.
Across 40 B2B SaaS audits run by Growth Engines in Q1 2026, brands with one clearly-defined definitional page per core feature earned roughly 31% higher AI citation rates than sites without them. A definitional page doesn't need to be long — it needs to answer "what is this thing?" in plain prose before doing anything else. Additionally, brands with five or more cross-platform mentions were cited by Perplexity 3.1× more often than brands with fewer than two.
Semrush data adds a useful negative signal: promotional language reduces AI citation rates by 26%. AI systems have learned to treat marketing register as a marker of lower reliability. The content that gets cited reads like reference material, not like a landing page.
The evidence above compresses into four concrete actions. Here they are in rough priority order:
First, verify that AI crawlers can reach your pages. Check robots.txt and confirm that GPTBot, PerplexityBot, and ClaudeBot are not blocked. Crawl accessibility is the prerequisite for everything else — no optimisation signal can work if the crawler never arrives.
Next, rewrite your three highest-traffic pages with answer-first H2s. The target structure: a direct, self-contained answer in the first 40–60 words under each heading. Write it so it makes sense without surrounding context. Answer-capsule structure is one of the most consistently cited content techniques across GEO research — it gives AI systems a citable unit without requiring them to interpret the whole page.
Then, add one cited statistic per 150–200 words across your pillar content. Link each statistic to its primary source, not a secondary summary. This is the ACM KDD finding in practice: the +41% visibility lift from adding statistics comes from the combination of specificity and verifiability.
Finally, track manually before you optimise further. Run monthly tests across ChatGPT, Perplexity, Claude, and Google AI Mode for your five core category queries. Screenshot and record which sources get cited. This baseline data is more valuable than any tool's aggregate dashboard, because it shows you exactly where you're absent and what the cited alternatives do differently.
On llms.txt itself: if your CMS generates it automatically (Mintlify, GitBook, Yoast), leave it running. The file has genuine value in agentic contexts and costs nothing when auto-generated. If it requires manual effort, redirect that time to the content structure work above. The studies have spoken. The ceiling on what the file can do for citation behaviour is, at current platform support levels, effectively zero.
Michael Shaskey
Founder · Klove
Prior to founding Klove, Michael spent over a decade at the intersection of engineering, growth marketing, and revenue operations — building GTM systems for high-growth B2B SaaS companies.
No. robots.txt is a well-established standard that all major search engines and AI crawlers recognise as an access-control signal. llms.txt, by contrast, is a proposed standard that no major AI provider — OpenAI, Anthropic, or Google — has documented as an input to their citation or crawling pipeline. Google's John Mueller has compared llms.txt to the deprecated meta keywords tag.
Not necessarily. If your CMS generates it automatically (Mintlify, GitBook, Yoast), keep it — the file has real value for agentic pipelines and costs nothing when auto-generated. However, if maintaining the file requires manual effort, that time is better spent on content structure improvements, which have measurable citation impact according to the ACM KDD 2024 study.
AI search citations happen when models like ChatGPT or Perplexity reference your content in their responses — driven mostly by training data and content quality signals. Agentic pipelines are different: they involve autonomous AI agents that browse and retrieve web content in real time on behalf of users. llms.txt reduces friction in agentic pipelines by providing curated Markdown, but it does not influence search citation behaviour.
No. Google has not indicated that llms.txt is used in AI Overviews or AI Mode. Google's Gary Illyes has stated on record that the company has "no plans to support LLMs.txt." AI Overviews rely on traditional search signals: E-E-A-T, backlink authority, content structure, and query relevance.
According to Aggarwal et al. (ACM KDD 2024, Princeton/Georgia Tech/IIT Delhi), citing authoritative external sources improved AI search visibility by 115% for lower-ranked content — the single largest lever across all techniques tested. Adding statistics lifted visibility by 41%, and adding quotations improved it by 28%.