Some AI services search through Brave: Lab Leaders' site had only 24% of its pages in the index
If an AI service looks up sources through the Brave search engine and your page is not in its database, the page may never make it into the answer, even if it ranks high on Google. Brave itself names Perplexity among the buyers of access to its index. The check takes minutes: the site: operator at search.brave.com, or a query to the API. Lab Leaders' site had only 24% of its pages in Brave's database (35 of 143), even though it had a sitemap, so Brave should be checked alongside Google.
AI-processed from Habr AI; edited by Hamidun News
Imagine this: you have spent years building your rankings in Google and Yandex, yet when a neural network answers your customer's question, it doesn't even see your article. The reason may be that the service looks for sources in a different search engine: Brave Search. This page database is sold to AI companies, and for one of the sites we checked, Lab Leaders, it held only 35 of 143 pages as of August 31, 2026, which is 24%. The links to those pages were in place, and the sitemap was served. The good news is that you can check your own site in a few minutes.
Why AI services use the Brave index
Brave Search maintains its own page database and sells access to it through the Search API: a program sends a query and receives the results in JSON format. An AI service needs this for grounding, that is, anchoring an answer in the pages it finds. The service searches for documents matching the user's question and builds the answer from them, adding to what the model already knows. Brave itself gives the index size: more than 40 billion pages in August 2026. There is no independent count.
Who exactly buys access? The About Brave page names Cohere, Mistral, Perplexity and You.com. In the New York Times lawsuit against Brave, filed on November 2, 2026, Anthropic, Perplexity, Mistral, xAI and Cohere are named, but none of them is a defendant. The newspaper claims that Brave copied publishers' content and sold it to AI companies. This is the plaintiff's position, and there is no court ruling. There is no confirmation from Perplexity, Mistral or Cohere themselves in open sources, and the services may have other search providers too. So the wording must be careful: a page that is not in Brave may not make it into these services' answers.
How to check your site manually and with a script
Manually, it is done at search.brave.com with the site: operator. The query site:example.com shows the domain's pages in the index, and it takes about 30 seconds. For a single page, you write site: followed by the full address with https, for example site:https://example.com/blog/page/. For a specific subdomain, write site:blog.example.com.
Two caveats. First, read the description under the link: Brave may show a cached copy, and the author of the Search Engine Journal review found his own site with an outdated homepage description. Second, an empty result proves nothing: Brave's documentation calls the search operators experimental. Before deciding that a page is missing, search for it with a regular query using its title.
For a list of addresses, the Brave Search API is more convenient. A Python script sends a GET request to api.search.brave.com/res/v1/web/search, puts site:address into the q parameter, passes the key in the X-Subscription-Token header, and checks whether the needed URL came back. The comparison is strict: only the anchor and a trailing slash are ignored. If Brave stores the page under a different address, it will get the status "not in results", and such rows need to be opened by hand. Keep a pause of 1.1 seconds between requests, and increase it if the API returns errors.
The economics are simple. Since February 2026 the Search plan has cost $5 per 1,000 requests. There is no free plan, but every plan includes $5 in credits per month, which is about 1,000 requests, provided you display Brave Search API as the source on your site. A site with 300 pages means 300 requests, which fits within the credits with room to spare.
One more technique: save which sources Perplexity cites in its answers to your customers' questions, and compare them with Brave's results. A match proves nothing, since another search engine may have found the same page. But it does show whether your pages are in the same places as the pages already being cited.
Why a page is missing from the index and what to do
First check for blocks. Brave's crawler has no name of its own in the access rules, so a separate robots.txt rule for it cannot be written. If a page is closed to Googlebot, Brave does not crawl it either. A noindex in the code or in the server response header removes a page from the index. The checklist is the same as for Google.
If there are no blocks, the pages may still be missing from the index. Brave's crawler partly draws on data from users of the Brave browser who have enabled the Web Discovery Project, and this option is off by default. Links and a sitemap do not guarantee inclusion, as the same 24% for Lab Leaders showed.
After removing any blocks, submit the addresses through the form at search.brave.com/submit-url. Brave's help pages mention it for re-crawling and removing pages. According to the Lab Leaders experience, the form is free, works without registration, and places addresses in a queue; it is better to submit them one at a time. Brave's documentation on accepting new addresses does not mention this. In the Lab Leaders case, indexing rose from 24% to 99% over ten weeks: as of November 9, that is 295 of 299 pages. The addresses were submitted through the form, and the index was checked every week.
Limitations
There was no control group, so it is unknown how much the address submission contributed and how much would have been indexed anyway. This is a single site. There are no open measurements of how completely Brave covers Russian-language sites. The link between AI services and Brave rests on Brave's own statements and on the plaintiff's lawsuit.
The practical conclusion: add Brave to your regular visibility checks alongside Google and Yandex. This takes minutes and a few dollars. If a site is not in the index, no amount of work on the texts will get it into answers built on that index.
- Nov 21, 2026A viral GTA 6 Claude clip looks fake: half an hour of control burned through the 5-hour Pro limit
- Nov 21, 2026Claude Code Speeds Up Snowboard Kids 2 Decompilation: From 20% to 45% of Functions in Three Months
- Nov 21, 2026FDE job postings grew 4.6x in a year: what a forward deployed engineer actually does at the client's site
- Nov 21, 2026Anthropic cuts Claude Fable 5.2 price by 20%, and on the coding test it nearly matches Opus 5.5
Key Questions Answered
Why AI services use the Brave index?
Brave Search maintains its own page database and sells access to it through the Search API: a program sends a query and receives the results in JSON format. An AI service needs this for grounding, that is, anchoring an answer in the pages it finds. The service searches for documents matching the user's question and builds the answer from them, adding to what the model already knows. Brave itself gives the index size: more than 40 billion pages in August 2026. There is no independent count. Who exactly buys access?
How to check your site manually and with a script?
Manually, it is done at search.brave.com with the site: operator. The query site:example.com shows the domain's pages in the index, and it takes about 30 seconds. For a single page, you write site: followed by the full address with https, for example site:https://example.com/blog/page/. For a specific subdomain, write site:blog.example.com. Two caveats.
Why a page is missing from the index and what to do?
First check for blocks. Brave's crawler has no name of its own in the access rules, so a separate robots.txt rule for it cannot be written. If a page is closed to Googlebot, Brave does not crawl it either. A noindex in the code or in the server response header removes a page from the index. The checklist is the same as for Google. If there are no blocks, the pages may still be missing from the index.
Test Frontier LLMs & Autonomous Agents in One Workspace
Run Claude Opus 5.5, GPT-6, Gemini 3.1, and DeepSeek side-by-side in Nomad AI (nomadnet.ai) — unified model routing, deep research, and multimodal workflows under one subscription.
The AI world, distilled — once a week
Seven stories that actually mattered, hand-picked. No noise, no reposts, no press releases.
Done! Check your inbox for a confirmation.