AI Crawlers See an Empty Shell Instead of Your Site: They Don't Execute JavaScript

The site ranks at the top of search, traffic keeps coming, conversions hold steady. Yet when asked "what is company X," ChatGPT names three competitors or repeats facts that are two years old. The authors of a recent Habr analysis ran their own site through queries to several models, looked at what those models' infrastructure receives, and reached a prosaic conclusion. There was enough content. It just never reached the model: some AI crawlers do not execute JavaScript, and they see an empty page shell instead of the article.

Why "more content" doesn't fix it

The first reaction from marketers and developers is predictable: rewrite the copy, add expertise, build out an FAQ. The hypothesis is logical, but in the authors' experience it almost never holds up. A neural network answers based not on what is written on your site, but on what reached it. If content gets lost between the server and the model, any edits to the text miss the target.

The main source of loss is simple. Modern sites built on React, Vue or Angular often serve the browser minimal HTML and scripts, and the text itself is assembled on the client side. A person sees the finished page. A crawler that does not run JavaScript gets `<div id="root"></div>` and nothing else.

Independent measurements confirm this. In a study by Vercel and MERJ from late 2024, researchers analyzed traffic from OpenAI and Anthropic bots: they requested JavaScript files but did not execute them. Googlebot, by contrast, renders pages fully, which is why a site can be fine for search and empty for a neural network.

Where content gets lost along the way

The path from a site to a model's answer has several links, and any of them can break:

• Rendering: the text appears only after JS runs, and the crawler does not run it.
• Access: robots.txt, a WAF or anti-bot protection blocks GPTBot, ClaudeBot, PerplexityBot and similar bots.
• Hidden content: key data sits in tabs, accordions and modal windows that load on click.
• Stale cache: the model relies on an old copy of the page or on third-party mentions rather than the current site.
• Weak structure: there are no headings, schema.org markup or update dates that would show what is fresh.

It is worth distinguishing training crawlers from search crawlers. The former collect data to train models; the latter pull in pages at the moment a user asks a question. Blocking the first should not cut off the second, yet at most companies the robots.txt rules are written as a single line covering all bots at once.

What to check and fix

You can start in fifteen minutes and without contractors.

1. Open the page source (view-source) or run `curl` and see whether the main text is in the HTML. If there are only scripts, you have found the problem. 2. Repeat the request with an AI bot's User-Agent. If the server returns a 403 or a captcha, access is closed at the protection level, not at the content level. 3. Check robots.txt and CDN rules: which bots are allowed and which are blocked on purpose. 4. Compare the answers of different models to questions about the company, and record which facts they know and which they get wrong.

If the text is missing from the HTML, the fix is server-side rendering (SSR) or static site generation (SSG): Next.js, Nuxt and Astro support both out of the box. For older projects, prerendering for known bots is sometimes enough. After that, it makes sense to move the essentials into plain HTML: product description, prices, contacts, update dates. Add structured markup and keep your sitemap current, with a correct lastmod.

The llms.txt format is sometimes proposed as a standalone solution. But there is still little reliable data on its real effect on model answers, and it should not be put ahead of basic content accessibility. First, make sure the page is readable without a browser. Everything else works only after that.