Crawling a knowledge base without a crawler service

Customers want the agent to answer from their own documentation, so I import their website: crawl the pages, pull out the useful bits, index them. For about a year I paid a crawling API to do it, per page.

Then I looked at what I was actually buying. A crawl is two jobs, not one. Finding out which pages exist, and turning each page into clean text. The first is almost free. The second is where all the money goes.

I had been paying the expensive price for both.

Splitting them took a fortnight and changed the economics completely, because once discovery is cheap you can show someone the list of pages first and let them approve the cost before you spend anything.

Most sites will just tell you their pages

You do not have to guess which pages exist. Sites publish that list themselves, and have done since long before anyone wanted to feed a website to a model.

My discovery route resolves in under two seconds and touches no browser and no LLM. It tries, in order:

1. robots.txt, parsed for Sitemap: directives, per RFC 9309.
2. /sitemap.xml, then /sitemap-index.xml, following one level of index nesting. The sitemap protocol defines both shapes.
3. If neither exists, one plain fetch of the page and a link harvest.

The fallback is where Workers gives you something unusual. HTMLRewriter is a streaming parser, so link extraction happens as bytes arrive without ever building a DOM.

const links = new Set<string>()

await new HTMLRewriter()
  .on('a[href]', {
    element(el) {
      const href = el.getAttribute('href')
      if (!href) return
      try {
        const url = new URL(href, base)
        if (url.hostname === new URL(base).hostname) links.add(url.href.split('#')[0])
      } catch {
        // relative junk, ignore
      }
    },
  })
  .transform(await fetch(base))
  .arrayBuffer()

Same-host filtering, a deny list for login and cart paths, dedupe, and a cap. Mine is 30 pages per import, which is not a technical limit. It is the number where the token estimate stays small enough that a user will approve it without thinking hard.

When do you actually need a browser?

When the HTML you get back has no content in it.

That is a smaller share of the web than the discussion around it suggests, but it is not small enough to ignore, and it is concentrated in exactly the documentation and help-centre sites people want to import. The test I use is crude and works: fetch plainly, extract text, and if the result is under a few hundred characters while the page clearly is not empty, escalate that URL to Browser Rendering.

Stage	Per page	What it gets you
Sitemap discovery	Negligible	The URL list
Plain fetch and parse	One subrequest	Text for most pages
Headless render	Browser session, seconds	Text for client-rendered pages
Model extraction	Input plus output tokens	Structured question and answer pairs

Reading down that table is the whole argument. Each row is roughly an order of magnitude more expensive than the one above it, so the job of the pipeline is to let as few pages as possible reach the bottom row.

I pool the browser sessions through a single Durable Object rather than opened per page. Sessions are the scarce resource, and one object owning them means the pool has a natural place to live and queue.

Estimate before you spend

The step that changed how people used this was showing the number first.

Discovery returns the URL list and a token estimate. The user sees “31 pages, roughly 51,000 tokens” and approves or trims the list. Nothing has been charged yet. Before the crawl starts I check the estimate against the tenant’s remaining allowance and refuse up front if it will not fit, rather than failing at page 22 with half an import written.

Refusing early is unglamorous and it removed an entire category of support ticket. Half-imported knowledge bases are miserable to clean up, because the user cannot tell which pages made it.

What I would do differently

Two things.

Conditional requests. I refetch pages I have already imported instead of sending If-None-Match and skipping unchanged ones. For a documentation site that is re-imported weekly, most pages have not moved.

And a slower crawl. I fire concurrent fetches at a stranger’s origin with no configured delay, which is rude and occasionally gets me rate limited. A one-second gap would cost me nothing on a 30-page import.

Neither is hard. Both are the kind of thing that only becomes visible once the expensive problem is solved and you start looking at what is left.

Sources

• HTMLRewriter, Cloudflare
• Browser Rendering, Cloudflare
• Sitemaps XML format, sitemaps.org
• RFC 9309: Robots Exclusion Protocol, IETF
• Workers AI, Cloudflare