feat(tools): add Cloudflare Browser Rendering backend for web_crawl#1008
Open
vu1n wants to merge 1 commit intoNousResearch:mainfrom
Open
feat(tools): add Cloudflare Browser Rendering backend for web_crawl#1008vu1n wants to merge 1 commit intoNousResearch:mainfrom
vu1n wants to merge 1 commit intoNousResearch:mainfrom
Conversation
vu1n
force-pushed
the
feat/web-crawl-cloudflare
branch
2 times, most recently
from
2f61681 to
c6639f1
Compare
Register web_crawl as a tool with support for two backends: - Firecrawl (existing) via the Firecrawl SDK - Cloudflare Browser Rendering (new) via the /crawl REST API Backend is auto-detected from env vars (CF_BROWSER_TOKEN + CF_ACCOUNT_ID for Cloudflare, FIRECRAWL_API_KEY for Firecrawl) or forced per-call via the `backend` parameter. WEB_CRAWL_BACKEND env var provides a global override. Cloudflare backend: POST to start an async crawl job, poll for completion, paginate results, normalize to the same page format as Firecrawl output.
vu1n
force-pushed
the
feat/web-crawl-cloudflare
branch
from
c6639f1 to
b55b469
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
web_crawlas a tool (previously defined but unregistered) with two backends: Firecrawl (existing) and Cloudflare Browser Rendering (new, per changelog)CF_BROWSER_TOKEN+CF_ACCOUNT_ID→ Cloudflare,FIRECRAWL_API_KEY→ Firecrawl), or forced per-call via thebackendparameterweb_crawlto thewebtoolset and_HERMES_CORE_TOOLSEnv vars
CF_ACCOUNT_IDCF_BROWSER_TOKENWEB_CRAWL_BACKENDcloudflareorfirecrawl)Test plan
uv run --extra dev pytest tests/tools/test_web_tools_config.py— 8/8 passweb_crawl, toolset resolution includes ithttpbin.org(2 pages crawled successfully)FIRECRAWL_API_KEY)Platform tested