# Crawling a knowledge base without a crawler service

Rendering every page in a headless browser is the expensive way to import a website. Discovery through robots.txt and sitemaps costs almost nothing, and most pages never need a browser at all.

By Elson Tan (https://elsontan.com)
Published: 2026-02-11
Reading time: 4 min
Tags: Cloudflare, Architecture, Cost
Canonical: https://elsontan.com/blog/crawling-without-a-crawler-service/

> Split website import into a cheap discovery pass and an expensive render pass. Sitemaps and a single streamed HTML parse enumerate URLs in under two seconds with no browser and no model. Only pages that return nothing useful get escalated to headless rendering, which is where the cost sits.

---
Customers want the agent to answer from their own documentation, so I import their website: crawl
the pages, pull out the useful bits, index them. For about a year I paid a crawling API to do it,
per page.

Then I looked at what I was actually buying. A crawl is two jobs, not one. Finding out which pages
exist, and turning each page into clean text. The first is almost free. The second is where all the
money goes.

I had been paying the expensive price for both.

Splitting them took a fortnight and changed the economics completely, because once discovery is
cheap you can show someone the list of pages first and let them approve the cost before you spend
anything.

## Most sites will just tell you their pages

You do not have to guess which pages exist. Sites publish that list themselves, and have done since
long before anyone wanted to feed a website to a model.

My discovery route resolves in under two seconds and touches no browser and no LLM. It tries, in
order:

1. `robots.txt`, parsed for `Sitemap:` directives, per [RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309).
2. `/sitemap.xml`, then `/sitemap-index.xml`, following one level of index nesting. The
   [sitemap protocol](https://www.sitemaps.org/protocol.html) defines both shapes.
3. If neither exists, one plain `fetch` of the page and a link harvest.

The fallback is where Workers gives you something unusual.
[HTMLRewriter](https://developers.cloudflare.com/workers/runtime-apis/html-rewriter/) is a streaming
parser, so link extraction happens as bytes arrive without ever building a DOM.

```ts
const links = new Set<string>()

await new HTMLRewriter()
  .on('a[href]', {
    element(el) {
      const href = el.getAttribute('href')
      if (!href) return
      try {
        const url = new URL(href, base)
        if (url.hostname === new URL(base).hostname) links.add(url.href.split('#')[0])
      } catch {
        // relative junk, ignore
      }
    },
  })
  .transform(await fetch(base))
  .arrayBuffer()
```

Same-host filtering, a deny list for login and cart paths, dedupe, and a cap. Mine is 30 pages per
import, which is not a technical limit. It is the number where the token estimate stays small
enough that a user will approve it without thinking hard.

## When do you actually need a browser?

When the HTML you get back has no content in it.

That is a smaller share of the web than the discussion around it suggests, but it is not small
enough to ignore, and it is concentrated in exactly the documentation and help-centre sites people
want to import. The test I use is crude and works: fetch plainly, extract text, and if the result
is under a few hundred characters while the page clearly is not empty, escalate that URL to
[Browser Rendering](https://developers.cloudflare.com/browser-rendering/).

| Stage | Per page | What it gets you |
| --- | --- | --- |
| Sitemap discovery | Negligible | The URL list |
| Plain fetch and parse | One subrequest | Text for most pages |
| Headless render | Browser session, seconds | Text for client-rendered pages |
| Model extraction | Input plus output tokens | Structured question and answer pairs |

Reading down that table is the whole argument. Each row is roughly an order of magnitude more
expensive than the one above it, so the job of the pipeline is to let as few pages as possible
reach the bottom row.

I pool the browser sessions through a single Durable Object rather than opened per page.
Sessions are the scarce resource, and one object owning them means the pool has a natural place to
live and queue.

## Estimate before you spend

The step that changed how people used this was showing the number first.

Discovery returns the URL list and a token estimate. The user sees "31 pages, roughly 51,000
tokens" and approves or trims the list. Nothing has been charged yet. Before the crawl starts I
check the estimate against the tenant's remaining allowance and refuse up front if it will not fit,
rather than failing at page 22 with half an import written.

Refusing early is unglamorous and it removed an entire category of support ticket. Half-imported
knowledge bases are miserable to clean up, because the user cannot tell which pages made it.

## What I would do differently

Two things.

Conditional requests. I refetch pages I have already imported instead of sending
`If-None-Match` and skipping unchanged ones. For a documentation site that is re-imported weekly,
most pages have not moved.

And a slower crawl. I fire concurrent fetches at a stranger's origin with no configured delay,
which is rude and occasionally gets me rate limited. A one-second gap would cost me nothing on a
30-page import.

Neither is hard. Both are the kind of thing that only becomes visible once the expensive problem is
solved and you start looking at what is left.

## Sources

- [HTMLRewriter](https://developers.cloudflare.com/workers/runtime-apis/html-rewriter/), Cloudflare
- [Browser Rendering](https://developers.cloudflare.com/browser-rendering/), Cloudflare
- [Sitemaps XML format](https://www.sitemaps.org/protocol.html), sitemaps.org
- [RFC 9309: Robots Exclusion Protocol](https://datatracker.ietf.org/doc/html/rfc9309), IETF
- [Workers AI](https://developers.cloudflare.com/workers-ai/), Cloudflare