scrapling-web-research

A Claude Code agent skill for web research using Scrapling — scrapes Reddit, Hacker News, ProductHunt, and Google Trends with anti-bot bypass.

What It Does

Provides structured web scraping tools for market research, startup idea validation, competitor analysis, and trend discovery. Handles bot protection automatically.

Source	Method	Anti-Bot
Reddit	JSON API (api.reddit.com)	Built-in
Hacker News	Scrapling Fetcher	Stealth headers
ProductHunt	Scrapling StealthyFetcher	Headless browser
Google Trends	Scrapling Fetcher	Stealth headers

Installation

As a Claude Code Skill

Add to your Claude Code settings (.claude/settings.json):

{
  "skills": [
    "github:YOUR_USERNAME/scrapling-web-research"
  ]
}

Manual Setup

git clone https://github.com/YOUR_USERNAME/scrapling-web-research.git
cd scrapling-web-research
bash reference/scripts/setup.sh

Usage

All-in-One Research

source /tmp/scrapling-research-env/bin/activate

python3 reference/scripts/research.py \
  --topic "AI agent marketplace" \
  --keywords "AI agent,hire AI,agent benchmark" \
  --sources reddit,hn \
  --format text

Reddit Only

python3 reference/scripts/reddit_scraper.py \
  --subreddits "SaaS,startups,Entrepreneur" \
  --query "AI startup idea" \
  --sort top \
  --timeframe month

Hacker News Only

python3 reference/scripts/hn_scraper.py \
  --pages front,show,ask \
  --days 3

ProductHunt Only

python3 reference/scripts/producthunt_scraper.py \
  --category "ai" \
  --limit 20

Output Formats

• --format text — Human-readable text (default)
• --format json — Structured JSON for further processing
• --output /path/to/file — Save to file instead of stdout

Requirements

• Python 3.10+
• Scrapling (auto-installed by setup script)

File Structure

scrapling-web-research/
  SKILL.md                              # Claude Code skill definition
  README.md                             # This file
  reference/
    scripts/
      setup.sh                          # One-time environment setup
      research.py                       # All-in-one research orchestrator
      reddit_scraper.py                 # Reddit JSON API + old.reddit fallback
      hn_scraper.py                     # Hacker News scraper
      producthunt_scraper.py            # ProductHunt with headless browser

License

MIT