restricted-web-research

A Claude / Agent Skill for web search and source extraction on machines with restricted egress — where the built-in WebSearch / WebFetch tools silently fail.

If you have ever seen this on an otherwise-online box:

Unable to verify if domain is safe to fetch.
This may be due to network restrictions or enterprise security policies.

…while curl https://example.com works fine from the same shell — this skill is for you.

The problem

Two independent failure modes commonly combine on locked-down hosts:

1. IPv6-first DNS + broken IPv6 egress. DNS returns AAAA (IPv6) records first, but the host has no IPv6 route / no global IPv6 address. Command-line curl survives because it implements Happy Eyeballs (falls back to IPv4); the agent tools' HTTP client does not fall back, so it errors out.
2. Outbound allowlist. Many destinations (Google, DuckDuckGo, Brave, Reddit, huggingface.co, github.com web/git) are dropped at the firewall and time out, while a handful (Bing, Marginalia, several JSON APIs, api.github.com) are reachable.

The net effect: the built-in web tools are unusable, but curl + a few local tools can still get the job done.

What's in here

File	Purpose
SKILL.md	The skill itself — reachable engines/APIs, recipes, and the research procedure.
scripts/probe-egress.sh	Diagnose your environment: proxy, DNS, IPv6-vs-IPv4, and which sources are reachable. Run this first — the reachability map in SKILL.md is an example from one environment; yours may differ.
scripts/bootstrap-tools.sh	Install the toolchain without root: bootstraps pip from a wheel, pip install --user for trafilatura/arxiv/uv, and apt-get download + dpkg-deb -x into ~/.local for jq/pandoc (with library-path wrapper scripts).

Quick start

git clone https://github.com/CaoCe123/restricted-web-research
cd restricted-web-research

# 1. Find out what your box can actually reach
bash scripts/probe-egress.sh

# 2. Install the toolchain (no sudo needed)
bash scripts/bootstrap-tools.sh

# 3. Install the skill for Claude Code (project- or user-level)
mkdir -p ~/.claude/skills/restricted-web-research
cp SKILL.md ~/.claude/skills/restricted-web-research/

Then in Claude Code, /restricted-web-research (or just ask it to search/research) will use curl + the local tools instead of the broken built-ins.

Toolchain (all installed under ~/.local, no root)

• trafilatura — HTML → clean markdown (primary extractor)
• pandoc — HTML/format → markdown fallback
• pdftotext — PDF deep-read (usually preinstalled)
• jq — JSON parsing
• uv / uvx, arxiv python lib — arXiv access

The apt-get download + dpkg-deb -x trick installs Debian/Ubuntu binaries into ~/.local without sudo (downloading and extracting do not need root); shipped wrapper scripts inject LD_LIBRARY_PATH / --data-dir so the binaries find their libs and data.

Caveats

• The reachability map is environment-specific — always re-run probe-egress.sh on a new host.
• Scraping search-engine HTML (Bing) is brittle and may break on markup changes or trigger captchas; the skill logs that case instead of returning empty.
• Be a good citizen: throttle APIs (arXiv rate-limits hard) and respect each site's terms.

License

MIT — see LICENSE.