Google’s llms.txt Guidance Depends On Which Product You Ask via @sejournal, @MattGSouthern

Google’s Search and Chrome documentation now point in different directions on llms.txt, depending on whether the goal is Search visibility or agentic browser readiness.

Google Search recently published a new optimization guide that lists llms.txt among the tactics you don’t need for generative AI features. The guide groups it with content chunking, AI-specific rewriting, and special schema.

Days earlier, Google’s Lighthouse tool shipped version 13.3, which added a new Agentic Browsing category. The update includes an llms.txt audit that checks whether a site provides the file and flags server errors when retrieving it.

The Lighthouse documentation describes llms.txt as a way to provide “a machine-readable summary of a website’s content, specifically designed for LLMs and AI agents.” It adds that without the file, “agents may spend more time crawling the site to understand its high-level structure and primary content.”

What Google Search Has Said

Google’s Search team has maintained for over a year that llms.txt is not a Google initiative or something Google plans to adopt.

John Mueller compared llms.txt to the keywords meta tag, noting no AI services used it and bots didn’t request the file. He called building separate Markdown pages for bots “a stupid idea.”

At Search Central Live Deep Dive Asia Pacific, Gary Illyes and Amir Taboul confirmed Google was not pursuing llms.txt.

Google’s optimization guide explicitly states llms.txt should be skipped, providing the most recent direct statement from the Search team.

What Chrome’s Lighthouse Now Does

Lighthouse 13.3 ships with the Agentic Browsing category by default, checking WebMCP integration, agent accessibility, layout stability, and llms.txt.

The llms.txt audit only marks sites as “Not Applicable” if they return a 404; errors flag the audit. The Lighthouse docs describe llms.txt as an “emerging convention” at llmstxt.org, advising site owners to create and place it in their root directory.

This category is separate from SEO audits and indicates that llms.txt helps browser-based agents understand site structure, not improve search rankings or AI citations.

Google Has Been Here Before

Google’s internal teams have sent mixed signals on llms.txt before.

In December, Lidia Infante spotted an llms.txt file on Google’s Search Central developer documentation. Mueller responded on Bluesky with “hmmn :-/” and didn’t clarify further.

Dave Smart noted that the file appeared on multiple Google developer properties, including developer.chrome.com and web.dev. The pattern suggested an internal CMS platform update that automatically deploys llms.txt files, not a Search team decision.

The Search Central file was removed within hours, but files on other Google properties remained.

Why This Matters

Google’s answer on llms.txt varies by use case.

For Google Search, llms.txt isn’t needed for AI Overviews, AI Mode, or other generative AI Search features.

For browser-based agents, Lighthouse considers llms.txt optional in an experimental machine interaction category.

Guidance is split between different Google developer sites, which can lead to conflicting instructions when comparing Lighthouse or its llms.txt documentation with Google’s Search docs.

Looking Ahead

Google hasn’t commented on the documentation gap between the two product teams.

For many sites, creating a basic llms.txt file is simple, but maintaining it is questionable, given that Google Search states it’s unnecessary for AI Search visibility.

Featured Image: Stock-Asso/Shutterstock