🕷️ Universal Web Scraper — full-site crawler fleet

Crawl entire websites — every page, every link — and extract all text, metadata, links, buttons, forms and tables. Built for speed (parallel crawlers + concurrent fetching), resilience (watchdog + multi-method retry for blocked pages), and privacy (runs 100% on your machine; cloud upload is optional).

✨ Features

• 🌐 Full-site crawling — follows every internal link up to unlimited depth
• ⚡ Concurrent engine — each crawler fetches multiple pages in parallel
• 🔀 Parallel fleet — spawn_workers.py auto-splits your site list across many crawlers
• 🛡️ Anti-block retry ladder — Chrome/Firefox/Safari browser impersonation → plain requests → text-proxy fallback
• ⏱️ Request watchdog — hung connections are abandoned after 60s, never freezing a crawl
• 🧾 Everything extracted — title, meta/OG tags, headings, paragraphs, tables, all links, buttons, forms
• ☁️ Optional cloud sync — point it at an rclone remote (Google Drive etc.), everything uploads automatically
• 🔁 Failure recovery — failed URLs are journaled and re-attempted by an aggressive retry pass
• ♻️ Resumable — stop anytime (Ctrl+C or fleet.sh stop); restart picks up exactly where it stopped

🚀 Quick start

pip3 install -r requirements.txt
pip3 install curl_cffi          # recommended: bypasses most bot-blockers

echo "https://example.com" > links.txt   # your target sites, one per line

python3 scraper.py                       # scrape everything

Results land in output/: one .json (full structured data) and one .txt (clean readable text) per site.

🎮 Fleet manager

For big lists, use the built-in manager:

chmod +x fleet.sh
./fleet.sh start    # launch main crawler + status loop + parallel workers
./fleet.sh status   # live snapshot: sites done, pages, pace, ETA
./fleet.sh bug      # scan & AUTO-FIX crashes, hangs, bad indexes
./fleet.sh stop     # stop safely, keep everything resumable

⚙️ Configuration (environment variables)

Variable	Default	Purpose
SCRAPER_CONCURRENCY	1	Parallel page fetches per crawler (try 6)
SCRAPER_DELAY	0.7	Politeness delay between batches (seconds)
SCRAPER_LINKS	links.txt	Your URL list
SCRAPER_STAGING	output	Local staging folder
SCRAPER_REMOTE	(unset)	rclone remote name → enables cloud sync
SCRAPER_FOLDER	scraped-data	Folder path on the remote

📂 Output per website

• <site>.json — structured data: full text, headings, paragraphs, tables, all internal/external links, buttons, forms, metadata (Open Graph etc.)
• <site>.txt — clean human-readable text of every page
• _REPORT.txt — accounting: scraped / recovered / permanently unreachable

☁️ Optional Google Drive sync (rclone)

rclone config                     # create a remote (e.g. named "mygdrive")
export SCRAPER_REMOTE=mygdrive
export SCRAPER_FOLDER=my-scraper-data

Now every finished site uploads automatically and local copies are cleaned. No rclone? Everything just stays in local output/.

⚠️ Responsible use

• Check each site's robots.txt and terms of service
• Keep politeness delays enabled (REQUEST_DELAY)
• Only scrape publicly available content

📜 License

MIT — do whatever you like, no warranty.