🕸️ CrawlKit – Your Friendly Internal Web Crawler in Go

CrawlKit is a fast and lightweight web crawler written in Golang.
It’s designed to explore links within a single website while giving you control over how many pages to crawl and how many requests to make at the same time.

Think of it as your personal web-surfing assistant — but for internal links only! 🌐

---

🚀 Features

• 🌍 Crawls only internal pages on the same domain
• 🔗 Handles both relative and absolute URLs automatically
• ⚡ Lets you configure concurrency (how many pages at once) and maximum page limits
• 📊 Generates a clean summary report showing which pages were found and how often
• 🧪 Comes with unit tests for HTML parsing, URL normalization, and reporting

---

🧩 Project Structure

Here’s how the project is organized:

.
├── main.go                       # CLI entry point
├── configure.go                   # Config and setup
├── crawlPage.go                   # Crawling logic
├── internal
│   ├── htmlparser                 # HTML parsing and link extraction
│   │   ├── fetchHTML.go
│   │   ├── extractLinks.go
│   │   └── extractLinks_test.go
│   ├── report                     # Report generation utilities
│   │   ├── generateReport.go
│   │   └── generateReport_test.go
│   └── urlutils                   # URL cleaning and normalization
│       ├── normalize.go
│       └── normalize_test.go
├── go.mod
└── go.sum

⚙️ Getting Started

Clone the repository and install dependencies:

git clone https://github.com/yourusername/crawlkit.git
cd crawlkit
go mod tidy

🧪 Running Tests

Make sure everything is working as expected:

go test ./...

🏗️ Building the Project

Build a binary so you can run the crawler without Go installed:

go build -o crawlkit

▶️ How to Use CrawlKit

Run the crawler from the command line like this:

go run main.go <base-url> [maxConcurrency] [maxPages]

Example

go run main.go https://golang.org 8 50

This will crawl up to 50 internal pages of https://golang.org using 8 workers at a time.

📋 Sample Output

Starting crawl for: https://golang.org
Concurrency: 8
Max pages: 50

Crawling page: https://golang.org/doc/
Crawling page: https://golang.org/pkg/
...

==========================
Crawl Summary – https://golang.org
==========================
Found 12 internal links to golang.org
Found 4 internal links to golang.org/doc
Found 2 internal links to golang.org/pkg
...