[Feature Request]: Lightweight HTTP-only crawling mode via DOM AST pruning (no browser required)

[dong7812]

What needs to be done?

Add an optional HTTP-only crawling strategy that fetches pages via plain HTTP and prunes the DOM AST directly — without launching a Playwright browser instance.

The strategy would:

1. Fetch raw HTML via aiohttp/httpx
2. Parse the DOM tree and strip non-content nodes (nav, header, footer, script, style, aside, ads) using a semantic tag allowlist
3. Apply BM25 section filtering (reusing the existing rank_bm25 infrastructure in content_filter_strategy.py)
4. Return clean Markdown — same output contract as today

New class: AsyncHTTPCrawlerStrategy as a sibling to the existing AsyncPlaywrightCrawlerStrategy

What problem does this solve?

For static and SSR-rendered pages (~60–70% of real-world crawl targets), spinning up a Playwright browser adds substantial overhead:

• 300–500ms cold start per page
• 200MB+ memory per browser process

There's currently no way to opt out of the browser layer even when JavaScript rendering isn't needed. This blocks Crawl4AI from being used in high-throughput, resource-constrained pipelines (serverless, edge, batch RAG ingestion).

Target users/beneficiaries

• RAG pipeline developers doing large-scale document ingestion
• Serverless / edge deployments where Playwright is too heavy to run
• Developers who only need to crawl static sites, documentation, blogs, or news
• MCP server authors embedding Crawl4AI as a lightweight tool

Current alternatives/workarounds

• Use requests + BeautifulSoup manually — loses Crawl4AI's BM25, output
  normalization, and async dispatcher
• Use AsyncPlaywrightCrawlerStrategy with headless=True and no JS — still pays
  the full browser startup cost
• Use a separate tool (e.g., dompruner-mcp) outside Crawl4AI's ecosystem

Benchmark from a reference implementation using the same DOM AST approach:

Page	Raw HTML	After pruning	Reduction
docs.python.org	44,316 tokens	1,328 tokens	97.0%
Average (50 sites)	baseline	—	93.5%

Parse time: ~11ms/page vs ~300ms Playwright cold start.
Reference: https://github.com/dong7812/dompruner-mcp

Proposed approach

1. New strategy class AsyncHTTPCrawlerStrategy in async_crawler_strategy.py

   • aiohttp for async HTTP (already a transitive dep)
   • lxml or html.parser for DOM tree traversal (no new deps)
   • Prune by tag allowlist identical to RelevantContentFilter.included_tags

2. Plug into the existing pipeline unchanged:

   • content_filter_strategy.py BM25 → no changes needed
   • markdown_generation_strategy.py → no changes needed
   • AsyncWebCrawler accepts strategy= parameter already

3. Opt-in via config:
   crawler_config = CrawlerRunConfig(crawler_strategy="http")

Foreseeable challenges:

• Pages that rely on client-side rendering will return empty/partial content
  → document clearly; auto-fallback to Playwright could be a follow-up
• robots.txt handling needs to be preserved (currently in Playwright strategy)

Success metric: same Markdown output quality as Playwright for static pages, with ≥10x reduction in memory footprint and ≥5x reduction in per-page latency.

Happy to prototype a PR targeting the develop branch.