Key takeaways
- I checked 200 B2B software company websites in November 2026: the Forbes Cloud 100 and 100 established B2B software companies.
- 112 of 200, or 56%, publish an llms.txt in the standard format. It was 56 in each group.
- None of the 186 robots.txt files I could read blocks an AI search crawler from the homepage. Two block GPTBot, ClaudeBot or Google-Extended. Eleven, or 5.9%, block at least one AI training crawler.
- 82% of readable homepages have structured data and 78% identify the company with Organization markup. Only 8% use FAQPage markup, and 37% have any question-style content at all.
- 22 of the 200 sites would not serve their homepage to a plain request with a browser user agent, and 6 more returned almost no text without JavaScript.
01The short answer
More than half of large B2B software companies now publish an llms.txt, and almost none of them block AI crawlers. The gap is on the page itself: most homepages tell a model who the company is, and very few answer a question a buyer would ask.
I wanted real numbers for this because the advice online runs ahead of the evidence. So I ran my AEO checker across 200 sites, then crawled each one a second time directly to confirm what it found.
02Results at a glance
| What I checked | Result | Out of |
|---|---|---|
| Publishes llms.txt in the standard format | 112 (56%) | 200 sites |
| Blocks an AI search crawler in robots.txt | 0 | 186 robots.txt files |
| Blocks GPTBot, ClaudeBot or Google-Extended | 2 (1.1%) | 186 robots.txt files |
| Blocks at least one AI training crawler | 11 (5.9%) | 186 robots.txt files |
| Names at least one AI crawler in robots.txt | 37 (19.9%) | 186 robots.txt files |
| Has any JSON-LD structured data | 146 (82%) | 178 readable homepages |
| Has Organization markup | 139 (78%) | 178 readable homepages |
| Links official profiles with sameAs | 127 (71%) | 178 readable homepages |
| Has SoftwareApplication or Product markup | 37 (21%) | 178 readable homepages |
| Has FAQPage markup | 15 (8%) | 178 readable homepages |
| Has question-style content on the homepage | 63 (37%) | 172 homepages the checker read |
| Has a comparison or pricing table on the homepage | 4 (2%) | 172 homepages the checker read |
| Under 150 words of text without JavaScript | 6 (3%) | 178 readable homepages |
| Refused or failed a plain request for the homepage | 22 (11%) | 200 sites |
03Almost nobody blocks AI crawlers
I expected a meaningful share of these companies to block AI crawlers in robots.txt. They don't. Of 186 robots.txt files I could read, not one blocks an AI search crawler from the homepage.
| Crawler | What it's for | Sites blocking it | Sites naming it |
|---|---|---|---|
| OAI-SearchBot | ChatGPT search | 0 | 23 |
| ChatGPT-User | Pages ChatGPT opens for a user | 0 | 23 |
| PerplexityBot | Perplexity search | 0 | 27 |
| Claude-SearchBot | Claude search | 0 | 11 |
| GPTBot | OpenAI model training | 1 | 24 |
| ClaudeBot | Anthropic model training | 1 | 26 |
| Google-Extended | Google AI training | 1 | 21 |
| Applebot-Extended | Apple AI training | 2 | 12 |
| Amazonbot | Amazon | 2 | 10 |
| meta-externalagent | Meta AI | 5 | 12 |
| CCBot | Common Crawl | 6 | 17 |
| Bytespider | ByteDance | 6 | 9 |
Two things stand out. The crawlers that do get blocked are mostly the ones with no search product a buyer uses: Common Crawl, ByteDance and Meta. And when a company names GPTBot, ClaudeBot or PerplexityBot in its robots.txt, it's nearly always to allow them or to set rules for parts of the site. Four in five don't mention AI crawlers at all, which leaves them allowed by default.
If you sell software and you're blocking these crawlers, you're in a very small minority, and your competitors are in the answers you're missing from. More on that in how to rank in ChatGPT.
04llms.txt: 56% have one
An llms.txt is a plain text file at the root of a site that lists the pages a company most wants a language model to read. 112 of the 200 sites publish one in the standard format, which starts with a heading and a one-line summary. Another 14 return something else at that address, such as a plain list of links.
The split was identical across the two groups: 56 of the Forbes Cloud 100 and 56 of the 100 established companies. So this isn't only a habit of AI startups. Payroll, security and accounting software companies have one too.
Does it do anything? I haven't seen an AI search provider confirm that llms.txt affects what gets cited, and I wouldn't publish one expecting it to move rankings by itself. I'd still add it. It takes an hour, it can't hurt, and more than half of the companies your buyers compare you with already have one. Then I'd spend the real effort on the next section.
05Sites say who they are. Few answer a question.
Identity is well covered. 82% of the homepages I could read have structured data, 78% mark up the company as an Organization and 71% link their official profiles with sameAs. A model can work out who these companies are.
What's thin is anything a model could quote in reply to a buyer's question:
- 37% of homepages have any question-style content, such as a heading phrased as a question with an answer under it.
- 8% use FAQPage markup.
- 21% describe the product with SoftwareApplication or Product markup.
- 2% have a comparison or pricing table on the homepage.
This matches what I see in client work. When I measured where AI answers came from for one B2B product, the cited pages were the ones that answered a specific question in plain sentences: integration pages, comparison pages and pricing explanations. A homepage hero and three feature cards give a model nothing to lift. I wrote up the method in the AEO playbook.
06One in nine sites wouldn't answer a plain request
22 of the 200 sites refused or failed a direct request for their homepage, even with an ordinary browser user agent. Most returned a 403 or a 429. That's bot protection doing its job against a script, and it doesn't prove the real AI crawlers are refused, because a firewall can let verified crawlers through and still turn away everything else.
It's still worth checking on your own site. Assistants also open pages live when a user asks about them, and a firewall rule written to stop scrapers can catch those requests too. Your firewall logs will tell you. Separately, 6 of the 178 homepages that did answer returned under 150 words of text without JavaScript. A crawler that doesn't run scripts sees an empty page there.
07What I'd do with this on your site
- Confirm you're not the exception.Read your robots.txt for rules naming OAI-SearchBot, ChatGPT-User, PerplexityBot or Claude-SearchBot, and check your firewall's bot settings. The free AEO checker does the first pass in about ten seconds.
- Fetch your key pages without JavaScript.If the words aren't in the raw HTML, fix that before anything else.
- Publish an llms.txt.An hour of work. List your product, pricing, integration and comparison pages. My llms.txt generator writes the file in the standard format.
- Answer questions on the page.This is where the gap is. Write the questions buyers ask your sales team as headings, answer each in two or three plain sentences, and mark them up.
- Measure whether it worked.Run a fixed set of buyer prompts every week and count how often you're named. See how to measure AI visibility.
If you'd like this done for your site, that's my AEO consulting and SaaS SEO work.
08How I ran it, and the limits
- The sample. 200 homepages: the 100 companies on the Forbes Cloud 100 for 2025, plus 100 established B2B software companies I picked, most of them publicly traded. It's a sample of large, well-funded companies, so smaller companies will look different.
- Two passes. My AEO checker fetched each site from Cloudflare's network. A second script fetched the homepage, robots.txt and llms.txt directly from a home connection. Where a site refused one, I used the other, and each number above says how many sites it's out of.
- robots.txt. I tested whether each crawler may fetch the homepage. A site that blocks a crawler from a subfolder only isn't counted as blocking.
- llms.txt. Counted only if the file exists and starts with a heading, as the proposed format describes.
- Homepages only. A company with no questions on its homepage may answer plenty on other pages.
- One day. Checked in November 2026. Sites change.
The full per-site results are in this CSV file. If you use the numbers, please link back to this page.
FAQQuestions people ask
- How many B2B SaaS companies have an llms.txt file?
- In Farhan Munshi's November 2026 check of 200 B2B software company websites, 112 of them, or 56%, published an llms.txt in the standard format. The share was the same in both halves of the sample: 56 of the Forbes Cloud 100 and 56 of 100 established B2B software companies.
- Do B2B software companies block GPTBot or ChatGPT in robots.txt?
- Almost none do. In Farhan Munshi's check of 186 readable robots.txt files from large B2B software companies, none blocked an AI search crawler such as OAI-SearchBot, ChatGPT-User or PerplexityBot from the homepage. One blocked GPTBot, one blocked ClaudeBot, and 11, or 5.9%, blocked at least one AI training crawler, mostly CCBot, Bytespider or Meta's crawler.
- Does llms.txt help you show up in ChatGPT or Perplexity?
- I haven't seen an AI search provider confirm that llms.txt changes what gets cited, and I wouldn't expect it to fix a site that is hard to read or has nothing worth quoting. It's cheap to publish, and more than half of the large B2B software sites I checked have one. I'd treat it as housekeeping and spend the real effort on pages that answer buyer questions.
- What is the most common AI search gap on B2B SaaS websites?
- Question and answer content. In Farhan Munshi's check, 82% of readable homepages had structured data and 78% identified the company with Organization markup, but only 37% had any question-style content on the homepage and 8% used FAQPage markup. Most sites tell a model who they are and very few answer the questions a buyer would ask.
- Should I block AI crawlers on my company website?
- If you sell software and want buyers to find you, probably not. The large B2B software companies in this check overwhelmingly let AI crawlers in. Blocking a training crawler is a business choice about your content. Blocking a search crawler such as OAI-SearchBot or PerplexityBot removes you from the answers your buyers are reading.
- How can I check whether AI crawlers can read my site?
- Read your robots.txt for rules that name GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot, fetch your page without JavaScript and count the words that come back, and check your firewall's bot settings. Farhan Munshi's free AEO checker at farhanmunshi.co/aeo-checker runs those checks on any URL in about ten seconds.
