fm.

llms.txt and AI crawler study: what 200 B2B software companies do

I ran my AEO checker and a second direct crawl across the Forbes Cloud 100 and 100 established B2B software companies. Here is who lets AI crawlers in, who publishes an llms.txt, and where almost everyone is still thin.

Key takeaways

  • I checked 200 B2B software company websites in November 2026: the Forbes Cloud 100 and 100 established B2B software companies.
  • 112 of 200, or 56%, publish an llms.txt in the standard format. It was 56 in each group.
  • None of the 186 robots.txt files I could read blocks an AI search crawler from the homepage. Two block GPTBot, ClaudeBot or Google-Extended. Eleven, or 5.9%, block at least one AI training crawler.
  • 82% of readable homepages have structured data and 78% identify the company with Organization markup. Only 8% use FAQPage markup, and 37% have any question-style content at all.
  • 22 of the 200 sites would not serve their homepage to a plain request with a browser user agent, and 6 more returned almost no text without JavaScript.

01The short answer

More than half of large B2B software companies now publish an llms.txt, and almost none of them block AI crawlers. The gap is on the page itself: most homepages tell a model who the company is, and very few answer a question a buyer would ask.

I wanted real numbers for this because the advice online runs ahead of the evidence. So I ran my AEO checker across 200 sites, then crawled each one a second time directly to confirm what it found.

02Results at a glance

What I checkedResultOut of
Publishes llms.txt in the standard format112 (56%)200 sites
Blocks an AI search crawler in robots.txt0186 robots.txt files
Blocks GPTBot, ClaudeBot or Google-Extended2 (1.1%)186 robots.txt files
Blocks at least one AI training crawler11 (5.9%)186 robots.txt files
Names at least one AI crawler in robots.txt37 (19.9%)186 robots.txt files
Has any JSON-LD structured data146 (82%)178 readable homepages
Has Organization markup139 (78%)178 readable homepages
Links official profiles with sameAs127 (71%)178 readable homepages
Has SoftwareApplication or Product markup37 (21%)178 readable homepages
Has FAQPage markup15 (8%)178 readable homepages
Has question-style content on the homepage63 (37%)172 homepages the checker read
Has a comparison or pricing table on the homepage4 (2%)172 homepages the checker read
Under 150 words of text without JavaScript6 (3%)178 readable homepages
Refused or failed a plain request for the homepage22 (11%)200 sites

03Almost nobody blocks AI crawlers

I expected a meaningful share of these companies to block AI crawlers in robots.txt. They don't. Of 186 robots.txt files I could read, not one blocks an AI search crawler from the homepage.

CrawlerWhat it's forSites blocking itSites naming it
OAI-SearchBotChatGPT search023
ChatGPT-UserPages ChatGPT opens for a user023
PerplexityBotPerplexity search027
Claude-SearchBotClaude search011
GPTBotOpenAI model training124
ClaudeBotAnthropic model training126
Google-ExtendedGoogle AI training121
Applebot-ExtendedApple AI training212
AmazonbotAmazon210
meta-externalagentMeta AI512
CCBotCommon Crawl617
BytespiderByteDance69

Two things stand out. The crawlers that do get blocked are mostly the ones with no search product a buyer uses: Common Crawl, ByteDance and Meta. And when a company names GPTBot, ClaudeBot or PerplexityBot in its robots.txt, it's nearly always to allow them or to set rules for parts of the site. Four in five don't mention AI crawlers at all, which leaves them allowed by default.

If you sell software and you're blocking these crawlers, you're in a very small minority, and your competitors are in the answers you're missing from. More on that in how to rank in ChatGPT.

04llms.txt: 56% have one

An llms.txt is a plain text file at the root of a site that lists the pages a company most wants a language model to read. 112 of the 200 sites publish one in the standard format, which starts with a heading and a one-line summary. Another 14 return something else at that address, such as a plain list of links.

The split was identical across the two groups: 56 of the Forbes Cloud 100 and 56 of the 100 established companies. So this isn't only a habit of AI startups. Payroll, security and accounting software companies have one too.

Does it do anything? I haven't seen an AI search provider confirm that llms.txt affects what gets cited, and I wouldn't publish one expecting it to move rankings by itself. I'd still add it. It takes an hour, it can't hurt, and more than half of the companies your buyers compare you with already have one. Then I'd spend the real effort on the next section.

05Sites say who they are. Few answer a question.

Identity is well covered. 82% of the homepages I could read have structured data, 78% mark up the company as an Organization and 71% link their official profiles with sameAs. A model can work out who these companies are.

What's thin is anything a model could quote in reply to a buyer's question:

  • 37% of homepages have any question-style content, such as a heading phrased as a question with an answer under it.
  • 8% use FAQPage markup.
  • 21% describe the product with SoftwareApplication or Product markup.
  • 2% have a comparison or pricing table on the homepage.

This matches what I see in client work. When I measured where AI answers came from for one B2B product, the cited pages were the ones that answered a specific question in plain sentences: integration pages, comparison pages and pricing explanations. A homepage hero and three feature cards give a model nothing to lift. I wrote up the method in the AEO playbook.

06One in nine sites wouldn't answer a plain request

22 of the 200 sites refused or failed a direct request for their homepage, even with an ordinary browser user agent. Most returned a 403 or a 429. That's bot protection doing its job against a script, and it doesn't prove the real AI crawlers are refused, because a firewall can let verified crawlers through and still turn away everything else.

It's still worth checking on your own site. Assistants also open pages live when a user asks about them, and a firewall rule written to stop scrapers can catch those requests too. Your firewall logs will tell you. Separately, 6 of the 178 homepages that did answer returned under 150 words of text without JavaScript. A crawler that doesn't run scripts sees an empty page there.

07What I'd do with this on your site

  1. Confirm you're not the exception.Read your robots.txt for rules naming OAI-SearchBot, ChatGPT-User, PerplexityBot or Claude-SearchBot, and check your firewall's bot settings. The free AEO checker does the first pass in about ten seconds.
  2. Fetch your key pages without JavaScript.If the words aren't in the raw HTML, fix that before anything else.
  3. Publish an llms.txt.An hour of work. List your product, pricing, integration and comparison pages. My llms.txt generator writes the file in the standard format.
  4. Answer questions on the page.This is where the gap is. Write the questions buyers ask your sales team as headings, answer each in two or three plain sentences, and mark them up.
  5. Measure whether it worked.Run a fixed set of buyer prompts every week and count how often you're named. See how to measure AI visibility.

If you'd like this done for your site, that's my AEO consulting and SaaS SEO work.

08How I ran it, and the limits

  • The sample. 200 homepages: the 100 companies on the Forbes Cloud 100 for 2025, plus 100 established B2B software companies I picked, most of them publicly traded. It's a sample of large, well-funded companies, so smaller companies will look different.
  • Two passes. My AEO checker fetched each site from Cloudflare's network. A second script fetched the homepage, robots.txt and llms.txt directly from a home connection. Where a site refused one, I used the other, and each number above says how many sites it's out of.
  • robots.txt. I tested whether each crawler may fetch the homepage. A site that blocks a crawler from a subfolder only isn't counted as blocking.
  • llms.txt. Counted only if the file exists and starts with a heading, as the proposed format describes.
  • Homepages only. A company with no questions on its homepage may answer plenty on other pages.
  • One day. Checked in November 2026. Sites change.

The full per-site results are in this CSV file. If you use the numbers, please link back to this page.

FAQQuestions people ask

How many B2B SaaS companies have an llms.txt file?
In Farhan Munshi's November 2026 check of 200 B2B software company websites, 112 of them, or 56%, published an llms.txt in the standard format. The share was the same in both halves of the sample: 56 of the Forbes Cloud 100 and 56 of 100 established B2B software companies.
Do B2B software companies block GPTBot or ChatGPT in robots.txt?
Almost none do. In Farhan Munshi's check of 186 readable robots.txt files from large B2B software companies, none blocked an AI search crawler such as OAI-SearchBot, ChatGPT-User or PerplexityBot from the homepage. One blocked GPTBot, one blocked ClaudeBot, and 11, or 5.9%, blocked at least one AI training crawler, mostly CCBot, Bytespider or Meta's crawler.
Does llms.txt help you show up in ChatGPT or Perplexity?
I haven't seen an AI search provider confirm that llms.txt changes what gets cited, and I wouldn't expect it to fix a site that is hard to read or has nothing worth quoting. It's cheap to publish, and more than half of the large B2B software sites I checked have one. I'd treat it as housekeeping and spend the real effort on pages that answer buyer questions.
What is the most common AI search gap on B2B SaaS websites?
Question and answer content. In Farhan Munshi's check, 82% of readable homepages had structured data and 78% identified the company with Organization markup, but only 37% had any question-style content on the homepage and 8% used FAQPage markup. Most sites tell a model who they are and very few answer the questions a buyer would ask.
Should I block AI crawlers on my company website?
If you sell software and want buyers to find you, probably not. The large B2B software companies in this check overwhelmingly let AI crawlers in. Blocking a training crawler is a business choice about your content. Blocking a search crawler such as OAI-SearchBot or PerplexityBot removes you from the answers your buyers are reading.
How can I check whether AI crawlers can read my site?
Read your robots.txt for rules that name GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, fetch your page without JavaScript and count the words that come back, and check your firewall's bot settings. Farhan Munshi's free AEO checker at farhanmunshi.co/aeo-checker runs those checks on any URL in about ten seconds.