Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ These options select which files the plugin reads and in what order.
| ---------------------- | ------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `docsDir` | `string \| DocsSection[]` | `'docs'` | Documentation source. A string base directory, or an array of section objects for multi-instance setups. See [Documentation sections](#documentation-sections). |
| `ignoreFiles` | `string[]` | `[]` | Glob patterns for files to skip. |
| `includeBlog` | `boolean` | `false` | Whether to include blog content. |
| `includeBlog` | `boolean` | `false` | Whether to include blog posts. See [include blog posts](content-generation.md#include-blog-posts). |
| `blogDir` | `string` | `'blog'` | Filesystem path to the blog content directory, relative to the site root. |
| `blogRouteBasePath` | `string` | `'blog'` | Docusaurus `routeBasePath` for the blog plugin, relative to the site `baseUrl`. Set this to match `presets.blog.routeBasePath` if you've customized it (for example `'news'`). |
| `includeOrder` | `string[]` | `[]` | Glob patterns controlling the order files are processed in. |
Expand Down Expand Up @@ -80,12 +80,12 @@ These options remove markup from the source Markdown before the plugin writes it

These options control the links the plugin emits.

| Option | Type | Default | Description |
| -------------------- | --------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `addMdExtension` | `boolean` | `true` | Append `.md` to link URLs in `llms.txt`, per the llmstxt.org spec. Only applies when `generateMarkdownFiles` is `true`. |
| `useRelativeUrls` | `boolean` | `false` | Emit links relative to the site origin (for example `/docs/page.md`). Useful for subpath deployments where the site `url` can't be pinned. |
| `rewriteImageUrls` | `boolean` | `false` | Rewrite relative image references to absolute hashed build-output URLs so LLMs can resolve them. |
| `pathTransformation` | `object` | `undefined` | Fallback path rewriting for URL construction. See [Path transformation](#path-transformation). |
| Option | Type | Default | Description |
| -------------------- | --------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `addMdExtension` | `boolean` | `true` | Append `.md` to link URLs in `llms.txt`, per the llmstxt.org spec. Only applies when `generateMarkdownFiles` is `true`. |
| `useRelativeUrls` | `boolean` | `false` | Write links as paths from the site root (for example `/docs/page.md`). See [relative links](ordering-and-paths.md#relative-links). |
| `rewriteImageUrls` | `boolean` | `false` | Rewrite relative image references to absolute hashed build-output URLs so LLMs can resolve them. |
| `pathTransformation` | `object` | `undefined` | Fallback path rewriting for URL construction. See [Path transformation](#path-transformation). |

The `pathTransformation` object accepts these fields:

Expand Down
20 changes: 20 additions & 0 deletions docs/content-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,26 @@ Version: 1.0.0
This file contains all documentation content in a single document following the llmstxt.org standard.
```

## Include blog posts

With `includeBlog: true`, the plugin adds the posts in the blog directory to `llms.txt` and `llms-full.txt`. `blogDir` (default `'blog'`) is the blog's folder, relative to the site directory, and `blogRouteBasePath` (default `'blog'`) is the blog plugin's `routeBasePath`, relative to the site `baseUrl`. Set both to match your blog plugin config:

```js
{
includeBlog: true,
blogDir: 'news',
blogRouteBasePath: 'news',
}
```

Each post links to the route Docusaurus builds for it:

- A post named with a date, such as `2024-01-01-hello.md` or `2024-01-01-hello/index.md`, links to `/blog/2024/01/01/hello`.
- A post with a front matter `slug` links to that slug under the blog route, such as `/blog/my-custom`.
- Posts with `draft: true` are skipped.

`ignoreFiles` and `includeOrder` patterns match blog posts by their site-relative path, such as `blog/2024-01-01-hello.md`. When `docsDir` lists several sections, the posts appear under a `## Blog` heading in `llms.txt`. In versions mode, posts appear in the version written at the site root only.

## How partials are resolved

The plugin supports [Docusaurus partials](https://docusaurus.io/docs/markdown-features/react#importing-markdown), the reusable MDX files imported into other documents. Partials are handled with no extra configuration:
Expand Down
25 changes: 24 additions & 1 deletion docs/ordering-and-paths.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: Set the order of documents with glob patterns, rewrite URLs for fil

# Ordering and path transformation

These options set the order documents appear in, rewrite the URLs of files that match no Docusaurus route, and set the batch size for progress logging. All of them are optional.
These options set the order documents appear in, write links as site-relative paths, rewrite the URLs of files that match no Docusaurus route, and set the batch size for progress logging. This page also covers how the plugin resolves each page's URL. All of them are optional.

For the full option list, see [configuration options](./configuration.md). For per-file variants of the ordering options, see [custom LLM files](./content-generation.md#generate-custom-llm-files).

Expand Down Expand Up @@ -70,6 +70,29 @@ Nesting depth follows the glob you write. `tutorials/beginner/**/*` matches begi

Custom LLM files have their own `orderPatterns` and `includeUnmatchedLast` fields. See [custom LLM files](./content-generation.md#generate-custom-llm-files) for details.

## How page URLs are resolved

The plugin links each page to the route Docusaurus built for it, which it receives through the `postBuild` hook. It derives the route from the page's file path and front matter the way Docusaurus does:

- Numeric ordering prefixes such as `01-` are dropped from every folder and file name, so `docs/01-guides/02-setup.md` links to `/docs/guides/setup`.
- `index.md`, `README.md`, and a file named after its folder (`api/api.md`), in any letter case, link to their folder's URL.
- A front matter `slug` that starts with `/` replaces the page's path under the section's `routeBasePath`: `slug: /start` in `docs/guides/intro.md` links to `/docs/start`.
- Any other `slug` replaces the file name and keeps the folder: `slug: setup` in `docs/01-guides/intro.md` links to `/docs/guides/setup`.
- A front matter `id` replaces the file name the same way, except on index, `README`, and folder-named files, where Docusaurus ignores it.
- A page matches routes under its own section's `routeBasePath` first, and docs pages never link to blog routes.

When `generateMarkdownFiles` is on, each `.md` file's path follows the same route, so `slug: /start` writes `docs/start.md`.

## Relative links

`useRelativeUrls` (type `boolean`, default `false`) writes each link in `llms.txt` and in links-only custom files as a path from the site root, keeping the `baseUrl`:

```
- [Intro](/sub/docs/intro): Docs intro.
```

Use it when the same build is served from more than one host, or when the site `url` in `docusaurus.config.js` isn't the URL readers use. Image URLs written by `rewriteImageUrls` stay absolute.

## Path transformation

The plugin resolves each document's URL by matching the file path against the routes Docusaurus built, which it receives through the `postBuild` hook. `pathTransformation` applies only to files that match no known route, so most sites don't need it.
Expand Down
Loading