Crawl4AI Cleans Web Pages for RAG and Caps Crawls at 25 Documents

RAG systems have an unpleasant characteristic: an expensive model will not help if menus, footers, repeated blocks, and irrelevant pages make their way into the knowledge base. A user asks about returning a product, but search retrieves the site's navigation and a list of related articles. The answer sounds confident, yet it is based on the wrong text.

AWS demonstrated a practical pipeline for such cases: Crawl4AI crawls approved pages, cleans them, saves the main content as Markdown, and adds metadata to each document. The files go to S3, while Amazon Bedrock Knowledge Bases handles chunking, embeddings, indexing, and retrieval.

When the built-in crawler is enough

AWS recommends starting with the standard Amazon Bedrock Web Crawler. You give it seed URLs or sitemaps, configure crawl boundaries and request frequency, and let it work. It respects robots.txt, can wait for JavaScript-loaded content, and, after the first full crawl, synchronizes only pages that were added, changed, or deleted.

This is a sensible option for documentation or reference websites where useful text is already clearly separated from the presentation layer. There is no need to deploy a custom crawler, maintain its state, or manage its scaling.

Crawl4AI comes into play when content must be controlled before ingestion. For example, a municipal service page may contain a header, a section directory, news, and just one block explaining the rules for obtaining a permit. The pipeline can select that block with a CSS selector, remove the remaining noise, and preserve headings, lists, and tables as Markdown. The source URL, crawl time, content type, and a custom category can be attached to the document; those fields can later be used to filter search results.

How the pipeline works

On a schedule, EventBridge starts an ECS task on Fargate. The task reads its settings from Systems Manager Parameter Store, crawls the configured websites, and validates boundaries both for the initial URLs and after redirects. Crawl scope can be restricted by domain, path, depth, URL patterns, and maximum page count. This prevents the crawler from unexpectedly moving into a neighboring section or attempting to collect the entire website.

Two files are created for every successfully processed page: a Markdown document and an adjacent file named page.md.metadata.json. This is the exact naming convention expected by the S3 data source in Bedrock. The metadata file is limited to 10 KB; the AWS example stops processing before writing either file if the limit is exceeded.

After the current document set has been uploaded, the task starts an ingestion job. Bedrock parses the new content, creates chunks and embeddings, and then updates the vector index. In a managed knowledge base, the service scales storage automatically.

What to check before production

The main trap is document chunking. In a managed knowledge base, the default mode creates 300-token chunks with 20% overlap. You can configure a different fixed size or disable chunking entirely for files prepared in advance. However, the selected strategy cannot be changed for an existing data source: experimentation requires creating a new one.

Start by selecting several representative pages and a set of real user questions. Compare the built-in Web Crawler with cleaned Markdown, then inspect the retrieved chunks and source links. Crawl4AI's heuristic filter can remove not only navigation but also a useful block, so a representative sample must be reviewed manually.

This example is designed for batch ingestion from known public websites, not for internet access while generating a response. Authenticated sections also require separate work on sessions, Secrets Manager, and security. The practical conclusion is straightforward: start with Bedrock's standard crawler and add Crawl4AI only when cleaning, precise extraction, and custom metadata measurably improve answers to your evaluation questions.