How to create a Webscraper integration (Chromium)

A webscraper tool launches a headless Chromium browser via Playwright, lets the page’s JavaScript run, and then hands the rendered HTML to the AI. Use this when the data only exists in the DOM after scripts have executed — React/Vue/Svelte apps, infinite-scroll pages, or anything that fetches its content client-side.

This is the slow path. Each call takes 2–8 seconds, costs more compute, and runs in an isolated browser environment where scrapes execute one at a time — so Chromium’s memory appetite never starves the rest of your project. Use it only when the parser genuinely won’t work.

When to use the scraper instead of the parser

Hit View Source on the page in your browser.
Search the source for the data you actually need.
Found it? → use Webparser, it’s ten times faster.
Empty page or just <div id="root"></div>? → you need the scraper.

Step 1 — Add a scrape tool with JS rendering

Same flow as the webparser — Admin → Projects → [project] → Integrations → Add tool — with two changes:

Type — Web Page Scrape.
Render JavaScript — turn it on. This is the only difference from a parser tool.
Allowed domains — still required. Strict allowlist: list exact hostnames, or add a wildcard entry like *.example.com — it covers example.com itself plus every subdomain.

Step 2 — CSS selector strategy

With a JavaScript app the page looks empty for a moment after load — then the framework paints the real content. Pass a CSS selector for the element you actually need; Playwright will wait for that selector to appear before grabbing the HTML. This is the difference between getting the data and getting the loading skeleton.

[data-testid="invoice-total"], .order-history-list

If your target loads progressively (lazy-loaded list, etc.), pick a selector that only exists once the data is in.

Step 3 — Custom headers and timeouts

Same as the parser: User-Agent, Cookie, or any other header you need is set under Custom headers. Cookies survive the navigation, so a header-based session token works fine.

The hard ceiling per call is 120 seconds, and slow scrapes start being wound down at around 90 — if your page genuinely takes longer than that, model the data behind a real API instead.

Step 4 — Test scrape

Use the same Test scrape button as the parser — the response shape is identical. The first call may be slower than steady-state because a fresh browser session has to spin up.

Production limits and gotchas

One scrape at a time. Scrape calls run sequentially in an isolated browser environment, and browser processes are recycled regularly to keep memory in check. Heavy concurrent traffic queues, it doesn’t parallelise.
Output cap is still 16 KB. Use a tight CSS selector to keep the AI’s context lean.
SSRF protection is on. Loopback, RFC 1918 ranges, and link-local are rejected before the browser launches.
Anti-bot pages will fight back. Cloudflare Turnstile, hCaptcha, and similar will render their challenge in place of your data — you can’t scrape past them. Reach out to the site owner for an API instead.
Don’t scrape your own internal apps just because it’s easier. A real API integration with bearer auth is faster, cheaper, more reliable, and survives UI redesigns.

Quick decision table

Server-rendered HTML → Webparser.
JS app, data appears after load → Webscraper.
Login-gated, no public data → API integration.
Static FAQ knowledge → KB article, no integration needed.