-brown?logo=linux&style=for-the-badge){kind=icon}
-blue?logo=apple&style=for-the-badge){kind=icon}
- Installation
- Running from source
- General Usage
- Supported sources
- Rejected sources
- Supported output formats
- Sponsors
This application uses Calibre to convert ebooks.
Install it from https://calibre-ebook.com/download
Without it, you will only get output in epub, text, and web formats.
For macOS, you need to manually add the path to Calibre.
Before starting lncrawl, use the command:
$ export PATH="$PATH:/Applications/calibre.app/Contents/MacOS"If you used a folder other than Applications during installation, replace /Applications/ with your path to Calibre.
⏬ Windows: lncrawl.exe
In Windows 8, 10 or later versions, it might say that
lncrawl.exeis not safe to download or execute. You should bypass/ignore this security check to execute this program.
⏬ Linux: lncrawl
It is recommended to install via pip if you are on Linux
⏬ MacOS: lncrawl
It is recommended to install via pip if you are on Mac
⏬ To get older versions visit the Releases page
📦 A python package named lightnovel-crawler is available at pypi.
Make sure you have installed Python v3.8 or higher and have pip enabled. Visit these links to install python with pip in Windows, Linux, and MacOS. Feel free to ask on the discussions page if you are stuck.
To install this app via pip:
$ pip install -U lightnovel-crawlerIn some cases you have to use python3 -m pip or pip3 or python -m pip.
Next, open your terminal and run:
$ lncrawl -h
If that does not work, you have not configured PATH variables properly. You can access it anyway using:
$ python3 -m lncrawl -h
The master branch contains the latest stable code. If you can not wait for it to be released in the PyPi, you can get it like this:
$ pip install -U git+https://github.com/lncrawl/lightnovel-crawler.git#egg=lightnovel-crawler
The dev branch contains cutting-edge, sometimes unstable changes. To install it:
$ pip install -U https://github.com/lncrawl/lightnovel-crawler/tarball/refs/heads/dev#egg=lightnovel-crawler
Docker is a convenient way to run it anywhere.
- Pull docker image:
$ docker pull ghcr.io/lncrawl/lightnovel-crawler
$ docker tag ghcr.io/lncrawl/lightnovel-crawler lncrawl
- Run using docker:
$ mkdir -p ~/.lncrawl
$ docker run -v ~/.lncrawl:/data --rm -it lncrawl
You can setup alias to the above command in your terminal's profile. e.g.:
alias docker-lncrawl=docker run -v ~/.lncrawl:/data --rm -it -p 8080:8080 lncrawl
- Run server with docker:
$ docker-lncrawl server
- First clone the repository:
$ git clone https://github.com/lncrawl/lightnovel-crawler- Open command prompt inside of the project folder and install requirements:
$ pip install -r requirements.txt- Run the program (use python v3.8 or higher):
$ python lncrawl- Build docker from source:
$ docker build -t lncrawl -f Dockerfile .
- Run server using docker:
$ docker compose -f ./scripts/local-compose.yml up
$ lncrawl -h
Usage: lncrawl [OPTIONS] COMMAND [ARGS]...
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --verbose -l Log levels: -l = warn, -ll = info, -lll │
│ = debug │
│ --config -c PATH Config file │
│ --install-completion Install completion for the current │
│ shell. │
│ --show-completion Show completion for the current shell, │
│ to copy it or customize the │
│ installation. │
│ --help -h Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ version Show current version. │
│ config View and modify configuration settings. │
│ sources Manage sources. │
│ crawl Crawl from novel page URL. │
│ search Search for novels by query string. │
│ server Run web server. │
╰──────────────────────────────────────────────────────────────────────────────╯
Run lncrawl server to start the server and visit http://localhost:8080.
$ lncrawl -ll server -p 8080
The default login for the web is:
username: admin
password: admin
For terminal usage, you can check help on each subcommands by -h:
$ lncrawl crawl -h
Usage: lncrawl crawl [OPTIONS] [URL]
Crawl from novel page URL.
╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────╮
│ url [URL] Novel details page URL. │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────╮
│ --noin Disable interactive mode │
│ --all Download all chapters │
│ --first N [x>=1] Download first few chapters │
│ --last N [x>=1] Download latest few chapters │
│ --format -f [json|epub|txt|pdf|mobi|docx|rtf|fb2|a Output formats │
│ zw3|lit|lrf|pdb|rb|tcr] │
│ --help -h Show this message and exit. │
╰───────────────────────────────────────────────────────────────────────────────────────────────────╯
Visit the discussions page for more information. You can also post your query there too.
Interested in contributing? Reference the
- CONTRIBUTING.md - Development setup and contribution guidelines
- docs/ARCHITECTURE.md - Codebase architecture overview
- docs/CREATING_CRAWLERS.md - How to add new source crawlers
- docs/DOCKER.md - Docker usage and deployment
- .github/FORKING.md - Setting up CI on forks
git clone https://github.com/lncrawl/lightnovel-crawler.git
cd lightnovel-crawler
make setup && make install
make startLightnovel Crawler Build System
Setup & Install:
make setup Create virtual environment
make install Install all dependencies (Python + Web)
make install-py Install Python dependencies only
make install-web Install web dependencies only
Development:
make start Start both backend and frontend servers
make start-server Start backend server only
make start-web Start frontend dev server only
make lint Run all linters
Build:
make build Full build (web + wheel + exe)
make build-web Build frontend only
make build-wheel Build Python wheel only
make build-exe Build PyInstaller executable
Docker:
make docker-build Build Docker image
make docker-up Start containers with docker compose
make docker-down Stop containers
make docker-logs View container logs
Other:
make version Show current version
make clean Remove build artifacts and venvs
Run the following command to add a new source:
$ lncrawl sources create
It can auto generate the code using ChatGPT.
Request new one by creating a new issue.
We are supporting 460 sources and 390 crawlers.
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍 | https://arnovel.me/ | 23 |   |
| 🔍 | https://kolnovel.com/ | 1 | |
| https://rewayat.club/ | 10 | |
| Source URL | Version | Contributors | |
|---|---|---|---|
| https://cclawtranslations.home.blog/ | 71 |  |
|
| https://domentranslations.wordpress.com/ | 71 | |
|
| 🔍🔑 | https://novelasligeras.net/ | 3 |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍🖼️ | https://anime-sama.fr/ | 4 |  |
| 🔍 | https://chireads.com/ | 10 | |
| 🔍 | https://lightnovelfr.com/ | 9 |  |
| 🤖🔍 | https://lnmtlfr.com/ | 9 | |
| 🤖🔍 | https://massnovel.fr/ | 1 | |
| https://noveldeglace.com/ | 3 | |
|
| https://xiaowaz.fr/ | 4 | |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🤖🔍 | https://ncode.syosetu.com/ | 19 |  |
| 🤖🔍 | https://syosetu.org/ | 2 |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍 | https://blnovels.net/ | 6 | |
| 🔍 | https://centralnovel.com/ | 13 | |
| 🔍 | https://illusia.com.br/ | 1 | |
| 🔍 | https://phoenixnovels.com.br/ | 1 |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🔍 | https://fenrirscans.com/ | 1 |
| Source URL | Version | Contributors | |
|---|---|---|---|
| 🤖🔍 | https://docln.net/ | 11 | |
| 🤖🔍 | https://ln.hako.vn/ | 11 | |
| 🤖🔍 | https://truyenfull.vn/ | 5 | |
| 🤖🔍 | https://truyentr.info/ | 5 | |
- JSON
- EPUB
- TEXT
- WEB
- DOCX
- MOBI
- RTF
- TXT
- AZW3
- FB2
- LIT
- LRF
- OEB
- PDB
- RB
- SNB
- TCR
 |
![@dependabot[bot]](https://avatars.githubusercontent.com/in/29110?s=64&v=4){kind=small}
