scrapling

scrapling

Scrapling as an iii worker. It maps Scrapling's three fetch tiers and its parsing engine to scrapling::* functions on the iii bus: fast HTTP with TLS impersonation, a Camoufox anti-bot browser, a full Playwright/Chromium browser, screenshots, and CSS/XPath/regex/adaptive extraction.

While connected it also injects a usage section into the agent system prompt via the harness pre-generate hook (scrapling::inject-guidance), so the guidance is presence-gated: no scrapling worker, no prompt text. The binding is one-shot at startup and relies on the engine's recoverable triggers (iii #1962, engine ≥ 0.21.8): bound before the harness is up, it parks as a pending intent and activates when the harness registers the trigger type. On older engines the bind is silently dropped.

Install

iii worker add scrapling

The worker is a deploy: image Python worker. The image build runs scrapling install, which downloads the Camoufox and Chromium browsers used by stealthy-fetch, dynamic-fetch, and screenshot.

Functions

Function	What it does
scrapling::fetch	HTTP get/post/put/delete (curl_cffi, TLS impersonation)
scrapling::stealthy-fetch	Camoufox stealth browser — Cloudflare/Turnstile bypass, WebRTC/canvas hardening
scrapling::dynamic-fetch	Playwright/Chromium — JS render, waits, XHR capture, CDP
scrapling::screenshot	Page screenshot (image content blocks) via a browser fetcher
scrapling::extract	Parse HTML with a declarative selector list
scrapling::css	One CSS query over provided HTML
scrapling::xpath	One XPath query over provided HTML
scrapling::regex	Regex over the visible text of provided HTML
scrapling::find-similar	An example element + structurally similar elements

Fetch output

Fetch functions return:

{ "status": 200, "url": "...", "headers": {}, "cookies": {}, "encoding": "utf-8",
  "extracted": { "...": "..." }, "html": "<...>" }

extracted appears only when selectors are given; html only when include_html: true. Called with a urls array, the response is { "results": [ ...one object per url... ] } (a failed URL yields { "url", "error" }, so one bad URL doesn't sink the batch).

Selector contract

selectors (on the fetchers and scrapling::extract) is a list of specs:

[
  { "name": "title", "css": "h1" },
  { "name": "links", "css": "a", "attr": "href", "all": true },
  { "name": "price", "regex": "price (\\d+)" },
  { "name": "card_html", "css": ".card", "html": true }
]

• one of css / xpath / regex per spec
• attr pulls an attribute; html pulls inner HTML; otherwise text
• all: true returns every match as a list, else the first match (or null)

Examples

# HTTP fetch + extract in one call
iii trigger scrapling::fetch --payload '{
  "url": "https://example.com",
  "selectors": [{ "name": "title", "css": "title" }]
}'

# Anti-bot page
iii trigger scrapling::stealthy-fetch --payload '{
  "url": "https://nopecha.com/demo/cloudflare",
  "solve_cloudflare": true,
  "selectors": [{ "name": "body", "css": "#padded_content a", "all": true }]
}'

# Parse HTML you already have
iii trigger scrapling::extract --payload '{
  "html": "<ul><li>a</li><li>b</li></ul>",
  "selectors": [{ "name": "items", "css": "li", "all": true }]
}'

# Fetch many URLs at once
iii trigger scrapling::fetch --payload '{ "urls": ["https://a.com", "https://b.com"] }'

Config

config.yaml holds operator defaults applied when a call omits the field:

defaults:
  impersonate: chrome     # HTTP fetcher fingerprint
  headless: true          # browser fetchers
  network_idle: false
  proxy: ""               # "" = none
  include_html: false
max_bulk_concurrency: 5

timeout is passed per call, not defaulted here — it means seconds for the HTTP fetcher and milliseconds for the browser fetchers.

How it maps

Scrapling	iii
Fetcher.{get,post,put,delete}	scrapling::fetch
StealthyFetcher.fetch	scrapling::stealthy-fetch
DynamicFetcher.fetch	scrapling::dynamic-fetch
browser page.screenshot()	scrapling::screenshot
Selector.css / .xpath / .re / .find_similar	extract / css / xpath / regex / find-similar

Boundaries

• Non-JSON Scrapling options (Python page_action/page_setup callbacks, proxy rotators, persistent sessions, the Spider crawl layer) are not exposed. Pass a single proxy string.
• The fetch functions are not agent-callable without human approval (outbound requests to arbitrary URLs); the pure parsers are. See iii-permissions.yaml.