How to create a Webscraper integration (Chromium)

A webscraper tool launches a headless Chromium browser via Playwright, lets the page’s JavaScript run, and then hands the rendered HTML to the AI. Use this when the data only exists in the DOM after scripts have executed — React/Vue/Svelte apps, infinite-scroll pages, or anything that fetches its content client-side.

This is the slow path. Each call takes 2–8 seconds, costs more compute, and runs on a dedicated celery-scrape worker (single-concurrency, isolated to keep Chromium’s memory footprint from starving the rest of the system). Use it only when the parser genuinely won’t work.

When to use the scraper instead of the parser

Hit View Source on the page in your browser.
Search the source for the data you actually need.
Found it? → use Webparser, it’s ten times faster.
Empty page or just <div id="root"></div>? → you need the scraper.

Step 1 — Add a scrape tool with JS rendering

Same flow as the webparser — Admin → Projects → [project] → Integrations → Add tool — with two changes:

Type — scrape.
Render JS — turn it on. This is the only difference from a parser tool.
Allowed domains — still required. Strict allowlist, no wildcards.

Step 2 — CSS selector strategy

With a JavaScript app the page looks empty for a moment after load — then the framework paints the real content. Pass a CSS selector for the element you actually need; Playwright will wait for that selector to appear before grabbing the HTML. This is the difference between getting the data and getting the loading skeleton.

[data-testid="invoice-total"], .order-history-list

If your target loads progressively (lazy-loaded list, etc.), pick a selector that only exists once the data is in.

Step 3 — Custom headers and timeouts

Same as the parser: User-Agent, Cookie, or any other header you need is set under Custom headers. Cookies survive the navigation, so a header-based session token works fine.

The hard ceiling per call is 120 seconds with a 90-second soft timeout — if your page genuinely takes longer than that, model the data behind a real API instead.

Step 4 — Test scrape

Use the same Test scrape button as the parser — the response shape is identical. The first call may be slower than steady-state because the worker spins up a new browser context.

Production limits and gotchas

One scrape at a time per worker. The celery-scrape queue is pinned to concurrency=1 with max-tasks-per-child=5 to recycle leaking browser processes. Heavy concurrent traffic queues, it doesn’t parallelise.
Output cap is still 16 KB. Use a tight CSS selector to keep the AI’s context lean.
SSRF protection is on. Loopback, RFC 1918 ranges, and link-local are rejected before the browser launches.
Anti-bot pages will fight back. Cloudflare Turnstile, hCaptcha, and similar will render their challenge in place of your data — you can’t scrape past them. Reach out to the site owner for an API instead.
Don’t scrape your own internal apps just because it’s easier. A real API integration with bearer auth is faster, cheaper, more reliable, and survives UI redesigns.

Quick decision table

Server-rendered HTML → Webparser.
JS app, data appears after load → Webscraper.
Login-gated, no public data → API integration.
Static FAQ knowledge → KB article, no integration needed.