TorSearch (Go MVP)

Minimal Tor search engine scaffold in Go with an AI-assisted crawler.

What is implemented

• Tor SOCKS5 fetcher (127.0.0.1:9050 by default)
• Onion version detection and separation:
  • v2: 16 chars + .onion
  • v3: 56 chars + .onion
• Separate crawl queues for v2/v3 with independent worker and page limits
• HTML parsing, onion link discovery, and crawl frontier dedupe
• In-memory full-text index split by onion version
• Heuristic AI enrichment (summary, tags, quality score)
• HTTP API:
  • GET /health
  • GET /stats
  • GET /search?q=...&version=v2|v3&limit=20

Important note about v2

Tor v2 onion services were deprecated and disabled in modern Tor around October 2021. This crawler can classify and queue v2 links, but successful fetching usually requires legacy-compatible environments.

Run

1. Start Tor locally (SOCKS5 at 127.0.0.1:9050).
2. Either set explicit seeds or configure seed source pages.
3. Run:

$env:TORSEARCH_SEEDS="aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.onion"
go run ./cmd/torsearch

Discover seeds from search pages

If TORSEARCH_SEEDS is empty, startup can fetch configured source pages and extract .onion URLs:

$env:TORSEARCH_SEED_SOURCES="https://your-search-page-1.example/?q=onion,https://your-search-page-2.example/onions"
$env:TORSEARCH_SEED_SOURCE_LIMIT="200"
go run ./cmd/torsearch

The discovered results are normalized to host-level seeds like http://<onion-host>/.

Run With Docker

1. Copy .env.example to .env.
2. Set at least one of TORSEARCH_SEEDS or TORSEARCH_SEED_SOURCES in .env.
3. Start the stack:

docker compose up -d --build

4. Check status:

docker compose ps
curl http://127.0.0.1:8080/health
curl http://127.0.0.1:8080/stats

If port 8080 is already in use, set TORSEARCH_HOST_PORT in .env (for example 18080) and query that port instead.

5. Stop:

docker compose down

Environment variables

Set at least one of TORSEARCH_SEEDS or TORSEARCH_SEED_SOURCES.

• TORSEARCH_SEEDS (optional): comma-separated onion seeds
• TORSEARCH_SEED_SOURCES (optional): comma-separated pages to scrape for onion seeds when TORSEARCH_SEEDS is empty
• TORSEARCH_SEED_SOURCE_LIMIT (default 200): maximum seeds discovered at startup
• TORSEARCH_HOST_PORT (default 8080): host port mapped to container port 8080
• TORSEARCH_HTTP_BIND (default :8080)
• TORSEARCH_TOR_SOCKS (default 127.0.0.1:9050)
• TORSEARCH_USER_AGENT (default torsearch-crawler/0.1)
• TORSEARCH_REQUEST_TIMEOUT (default 30s)
• TORSEARCH_MAX_BODY_BYTES (default 2097152)
• TORSEARCH_WORKERS_V2 (default 1)
• TORSEARCH_WORKERS_V3 (default 4)
• TORSEARCH_MAX_PAGES_V2 (default 0)
• TORSEARCH_MAX_PAGES_V3 (default 1000)
• TORSEARCH_MAX_DEPTH (default 2)
• TORSEARCH_RETRY_LIMIT (default 2)
• TORSEARCH_RETRY_BACKOFF (default 3s)
• TORSEARCH_IDLE_SLEEP (default 350ms)

Next production steps

• Persist crawl state/frontier in PostgreSQL or Redis
• Replace in-memory index with OpenSearch/Meilisearch
• Add robust duplicate detection and content safety controls
• Add distributed workers and observability