Law firms blocking AI crawlers: 1 in 8 California injury firm websites turn AI away
Key findings: how many law firms are blocking AI crawlers
We crawled the websites of personal injury firms that show up in Google Maps across 20 California metros and checked whether the AI assistants people now use to find a lawyer can actually read them. Of 816 sites we could load, 101 (12.4%) block at least one AI assistant's crawler. Most of them almost certainly do not know.
Most blocks are invisible to the firm
The usual way to block a crawler is a line in robots.txt, a public file anyone can read. Only 28 firms (3.4%) do that. The other 73 of the 101 firms block AI crawlers at the server, hosting or firewall level: their robots.txt allows the bot, but the server answers it with "access denied" (HTTP 403) while a normal browser gets the page. That usually comes from a security setting at the host or CDN, such as a bot-blocking toggle, a web application firewall rule or a security plugin, not from a decision anyone at the firm made.
This matters because the firm, its web designer and most SEO tools check robots.txt and see nothing wrong. Meanwhile, the assistant that a potential client asks for a lawyer cannot read the firm's practice pages, results or reviews on its own site.
Anthropic's ClaudeBot was refused most often: 93 firms, against 58 for OpenAI's GPTBot and 40 for PerplexityBot. Several firewall rules we saw block ClaudeBot by name while letting GPTBot through.
4 in 10 law firm websites do not say they are a law firm
Schema markup is code that tells search engines and AI assistants, in a structured way, what a page is about: that this is a law firm, where it is, what it practices and who the attorneys are. It is one of the easiest signals for an assistant to use. Here is what the homepages showed:
On the positive side, 48.4% use a legal schema type (LegalService, Attorney or LawFirm), and every readable site used HTTPS. 34.8% publish an llms.txt file, mostly generated automatically by SEO plugins. llms.txt has no proven effect on AI answers yet, so we report it but do not count its absence as a problem.
Does blocking AI relate to Google Maps rank?
We split the firms by the best Google Maps position we saw for "car accident lawyer", "personal injury lawyer" or "personal injury attorney" in their metro:
| Best Maps position | Firms | Block an AI crawler | Legal schema |
|---|---|---|---|
| Top 3 | 27 | 7% | 56% |
| 4 to 10 | 63 | 16% | 59% |
| 11 to 20 | 66 | 15% | 56% |
| 21 and below | 660 | 12% | 46% |
The top-3 group is small, so treat this as directional. It does suggest that firms near the top of the map pack are a bit less likely to have AI blocks and a bit more likely to use legal schema. Blocking is not limited to small or weak sites: firms with 50 or more pages blocked an AI crawler at 13%, more than firms with under 20 pages (8%).
Why law firms blocking AI crawlers is a problem
When someone asks ChatGPT, Claude, Perplexity or Google's AI for a car accident lawyer, the assistant searches the web and builds an answer from pages it can read. If it cannot read your site, it has to rely on what directories, reviews and other sites say about you, and it is more likely to name a competitor whose site it could read. PerplexityBot builds the index Perplexity answers from, so blocking it can keep a firm out of live answers, not only out of future training data. GPTBot and ClaudeBot are mainly training crawlers, but blocks rarely come alone: 41 of the 101 blocking firms refused all three.
Our guide on why ChatGPT recommends other law firms covers the other common reasons, and GEO for law firms explains the full approach.
How to check if your firm's website blocks AI crawlers
- Read your robots.txt. Open yourfirm.com/robots.txt and look for "Disallow: /" under GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot or PerplexityBot.
- Check the server, not just robots.txt. This is where most blocks were. Ask your host or web designer whether a bot-blocking or "block AI scrapers" setting is on at the CDN, firewall or security plugin, and have verified AI crawlers allowed.
- Test it. The free law firm SEO audit requests your homepage as GPTBot, ClaudeBot and PerplexityBot (twice each) and tells you which ones are turned away. The AI visibility audit also checks whether ChatGPT, Gemini, Claude, Perplexity and Google AI Overviews name your firm.
- Add the basics. LegalService or Attorney schema with your name, address and phone, and an FAQ section with FAQ schema on key pages. The law firm SEO audit checklist lists every check.
Methodology
Sample. We started from Google Maps results for "car accident lawyer", "personal injury lawyer" and "personal injury attorney" in 20 California metros (pulled October 23, 2026, up to 200 results per search). We kept firms categorized as law firms with at least 20 Google reviews, removed multi-office chains (a domain linked from three or more profiles) and non-PI practices. That left 1,048 firms and 976 unique websites. 160 sites timed out or refused an ordinary browser request from our server, so they are excluded, leaving 816 sites.
Checks. On November 10, 2026 we fetched each homepage, robots.txt and llms.txt with a normal browser user agent, then fetched the homepage again identifying as GPTBot, ClaudeBot and PerplexityBot. A server block counts only when the site returned 401, 403 or 406 (or a challenge page) to the AI user agent while returning the page to the browser, and did so again on a second, separate pass. Rate-limit responses (429) were not counted. robots.txt rules were evaluated with a standard robots.txt parser for each crawler. Schema types were read from JSON-LD on the homepage only.
Limits. Our requests came from our own server, not from OpenAI, Anthropic or Perplexity IP addresses. Some firewalls treat verified crawlers differently, so a refusal here is a strong signal, not proof, of what the real crawler gets. Homepage-only schema undercounts firms that use schema on inner pages. This is California personal injury firms only. We do not name any firm. Firms in the sample can run the free audit to see their own result.
You may cite these numbers with a link to this page. Reporters can reach Mark at mark@lawyerseoaudit.com.
Law firms blocking ai crawlers FAQ
Should a law firm block AI crawlers?
For most firms, no. Blocking AI search and answer crawlers such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot can keep your firm out of the answers people get when they ask an assistant for a lawyer. Some firms choose to block training-only crawlers for content reasons, which is a separate decision. Make it on purpose, not through a default security setting.
Why would a website block AI crawlers without the owner knowing?
Hosting companies, CDNs and security plugins increasingly include settings that block AI bots or unknown crawlers, sometimes turned on by default or during a security update. Because the block happens at the server, robots.txt still looks fine, so the firm and its designer have no reason to notice. In our data, 73 of the 101 blocking firms fell into this group.
Does blocking AI crawlers hurt Google rankings?
Blocking GPTBot, ClaudeBot or PerplexityBot does not affect Googlebot, so regular Google rankings are not directly hurt. Blocking Google-Extended affects how Google uses your content for Gemini, not search ranking. The bigger cost is in AI answers: ChatGPT, Claude and Perplexity cannot read your pages when they build a recommendation, so they lean on other sources.
Can I use these numbers in an article?
Yes. You may cite the figures with a link to this page as the source. The data covers 816 California personal injury firm websites checked on November 10, 2026, with the method described above. We do not publish firm-level results. Reporters who want more detail or a comment can email Mark Gundrum at mark@lawyerseoaudit.com.
Free audit for solo personal injury lawyers
The free audit checks your Google Maps rank around your office, whether ChatGPT, Gemini, Claude, Perplexity and Google's AI Overviews name you, and your website, in about a minute.
Prefer to talk? Book a 15-minute call with Mark.