feat(cli): support openkb add <URL> with content-type sniffing (#55)

* feat(cli): support 'openkb add <URL>' with content-type sniffing

`openkb add` now accepts http(s) URLs in addition to local paths.
URLs are fetched into raw/ first, then handed to the existing
add_single_file pipeline — so PageIndex / markitdown / hash dedup /
remove all work unchanged.

Routing is decided by the HTTP Content-Type header, validated
against magic bytes (so a CDN that mislabels a PDF as
'application/octet-stream' still routes correctly):

  PDF response   → urllib chunked download → raw/<sanitized>.pdf
                   → existing convert_document → PageIndex (≥20 pages)
                   or markitdown (short)

  HTML response  → trafilatura.fetch_url + extract main content
                   → raw/<title-slug>.md
                   → existing markitdown short-doc path

  Anything else  → clear error, returns None

Filename derivation: Content-Disposition header (RFC 5987 / quoted /
unquoted forms) → URL basename → URL host fallback. Sanitization
preserves identifier dots like arxiv's "2509.11420" (only strips an
extension when it matches the target extension), replaces shell-
unsafe chars (space / parens / etc.) with '-', collapses repeats,
and caps the stem at 80 chars.

Failure modes covered with stderr messages, no crashes:
  - HTTPError (404, 403, etc.) → "[ERROR] HTTP <code>"
  - URLError (DNS, refused)    → "[ERROR] Network error"
  - trafilatura returns None   → "[ERROR] No main content extracted"
  - <300 chars extracted       → "[WARN] only N chars — page may be JS-rendered"
                                  (still saved so user can inspect)
  - Unsupported content type   → "[ERROR] Unsupported content type"

Architecturally lives in openkb/url_ingest.py — peer to
openkb/converter.py and openkb/indexer.py rather than bloating
cli.py. cli.py only adds a 4-line intercept at the top of `add`.

Live-tested against four URL shapes:
  - arxiv.org/abs/<id>   → trafilatura → "Trading-R1-...md" (3 KB)
  - arxiv.org/pdf/<id>   → urllib      → "2509.11420v1.pdf" (2.1 MB)
                             (server's Content-Disposition gives the
                              version suffix for free — no special
                              arxiv handling needed)
  - claude.com/blog/...  → trafilatura → ".md" (1.5 KB)
  - CDN .pdf with %20    → urllib      → sanitized filename (465 KB)

31 new tests cover all routing branches, magic-byte override paths,
Content-Disposition parsing (3 forms), filename sanitization edge
cases (arxiv ID, spaces, long names, empty), and graceful failure
on HTTP/network errors. 360 total tests pass (329 prior + 31 new).

* fix(url-ingest): address PR #55 self-review findings

- _unique_path() prevents silent overwrites in raw/ when two URLs
  sanitize to the same filename (applied at both PDF and HTML write
  sites)
- PDF filename now derives from response.geturl() (the post-redirect
  URL), so DOI/shortlink resolvers produce sensible names
- add_single_file() returns bool; URL branch unlinks the fetched
  raw/* file when the downstream add is skipped (dedup), preventing
  orphan files that re-resurrect on every retry

* fix(url-ingest): preserve raw file on pipeline failure, fix HTML echo

Round-2 review on commit 47e5ec2 surfaced two issues:

1. add_single_file returned bool, so the URL branch couldn't tell
   "dedup skip" from "mid-pipeline failure". A transient LLM error
   during compile_long_doc would delete the just-downloaded PDF and
   leave a dangling PageIndex entry (hash registration only happens
   on full success, so `openkb remove` couldn't recover it). Now the
   function returns Literal["added","skipped","failed"]; the URL
   branch only unlinks on "skipped" — failures keep the raw file so
   the user can retry without re-downloading.

2. _extract_html echoed the pre-collision filename instead of
   target.name, so on a title collision the user was told the file
   was saved to a path occupied by the previous document. PDF branch
   already used target.name correctly; HTML branch now matches.

* chore(tests): drop unused 'from pathlib import Path' import

Flagged by github-code-quality bot on PR #55. The import was left over
from an earlier draft; nothing in the file references Path.

Author: KylinMountain

README.md

@@ -72,7 +72,8 @@ openkb init
 
 # 3. Add documents
 openkb add paper.pdf
-openkb add ~/papers/  # Add a whole directory
+openkb add ~/papers/                            # Add a whole directory
+openkb add https://arxiv.org/pdf/2509.11420     # Or fetch from a URL
 
 # 4. Ask a question
 openkb query "What are the main findings?"
@@ -148,7 +149,7 @@ A single source might touch 10-15 wiki pages. Knowledge accumulates: each docume
 | Command | Description |
 |---|---|
 | `openkb init` | Initialize a new knowledge base (interactive) |
-| <code>openkb&nbsp;add&nbsp;&lt;file_or_dir&gt;</code> | Add documents and compile to wiki |
+| <code>openkb&nbsp;add&nbsp;&lt;file_or_dir_or_URL&gt;</code> | Add documents and compile to wiki. URL ingest auto-detects PDF (saved as `.pdf` → PageIndex / markitdown) vs HTML (trafilatura main-content extract → `.md`) |
 | <code>openkb&nbsp;remove&nbsp;&lt;doc&gt;</code> | Remove a document and clean up its wiki pages, images, registry, and PageIndex state (use `--dry-run` to preview, `--keep-raw` / `--keep-empty-concepts` to retain artifacts) |
 | <code>openkb&nbsp;query&nbsp;"question"</code> | Ask a question over the knowledge base (use `--save` to save the answer to `wiki/explorations/`) |
 | `openkb chat` | Start an interactive multi-turn chat (use `--resume`, `--list`, `--delete` to manage sessions) |

openkb/cli.py

@@ -14,6 +14,7 @@
 import sys
 import time
 from pathlib import Path
+from typing import Literal
 
 import os
 
@@ -130,14 +131,22 @@ def _find_kb_dir(override: Path | None = None) -> Path | None:
     return None
 
 
-def add_single_file(file_path: Path, kb_dir: Path) -> None:
+def add_single_file(file_path: Path, kb_dir: Path) -> Literal["added", "skipped", "failed"]:
     """Convert, index, and compile a single document into the knowledge base.
 
     Steps:
     1. Load config to get the model name.
     2. Convert the document (hash-check; skip if already known).
     3. If long doc: run PageIndex then compile_long_doc.
     4. Else: compile_short_doc.
+
+    Returns:
+        ``"added"`` on full success, ``"skipped"`` when the file's hash
+        is already in the registry (dedup), or ``"failed"`` when any
+        pipeline stage raised. URL-ingest distinguishes these so it can
+        unlink the just-downloaded raw file on dedup (it would otherwise
+        be an orphan) while preserving it on failure so the user can
+        retry without re-downloading.
     """
     from openkb.agent.compiler import compile_long_doc, compile_short_doc
     from openkb.state import HashRegistry
@@ -156,11 +165,11 @@ def add_single_file(file_path: Path, kb_dir: Path) -> None:
     except Exception as exc:
         click.echo(f"  [ERROR] Conversion failed: {exc}")
         logger.debug("Conversion traceback:", exc_info=True)
-        return
+        return "failed"
 
     if result.skipped:
         click.echo(f"  [SKIP] Already in knowledge base: {file_path.name}")
-        return
+        return "skipped"
 
     doc_name = file_path.stem
     index_result = None  # populated only on the long-doc branch
@@ -174,7 +183,7 @@ def add_single_file(file_path: Path, kb_dir: Path) -> None:
         except Exception as exc:
             click.echo(f"  [ERROR] Indexing failed: {exc}")
             logger.debug("Indexing traceback:", exc_info=True)
-            return
+            return "failed"
 
         summary_path = kb_dir / "wiki" / "summaries" / f"{doc_name}.md"
         click.echo(f"  Compiling long doc (doc_id={index_result.doc_id})...")
@@ -192,7 +201,7 @@ def add_single_file(file_path: Path, kb_dir: Path) -> None:
                 else:
                     click.echo(f"  [ERROR] Compilation failed: {exc}")
                     logger.debug("Compilation traceback:", exc_info=True)
-                    return
+                    return "failed"
     else:
         click.echo(f"  Compiling short doc...")
         for attempt in range(2):
@@ -206,7 +215,7 @@ def add_single_file(file_path: Path, kb_dir: Path) -> None:
                 else:
                     click.echo(f"  [ERROR] Compilation failed: {exc}")
                     logger.debug("Compilation traceback:", exc_info=True)
-                    return
+                    return "failed"
 
     # Register hash only after successful compilation
     if result.file_hash:
@@ -225,6 +234,7 @@ def add_single_file(file_path: Path, kb_dir: Path) -> None:
 
     append_log(kb_dir / "wiki", "ingest", file_path.name)
     click.echo(f"  [OK] {file_path.name} added to knowledge base.")
+    return "added"
 
 
 # ---------------------------------------------------------------------------
@@ -395,12 +405,38 @@ def init(language):
 @click.argument("path")
 @click.pass_context
 def add(ctx, path):
-    """Add a document or directory of documents at PATH to the knowledge base."""
+    """Add a document or directory of documents at PATH to the knowledge base.
+
+    PATH may be a local file, a local directory (which is walked
+    recursively for supported extensions), or an http(s) URL. URLs are
+    fetched into ``raw/`` first: PDF responses (by Content-Type and
+    magic-byte sniff) are saved as ``.pdf``; HTML responses are run
+    through trafilatura's main-content extractor and saved as ``.md``.
+    """
     kb_dir = _find_kb_dir(ctx.obj.get("kb_dir_override"))
     if kb_dir is None:
         click.echo("No knowledge base found. Run `openkb init` first.")
         return
 
+    # URL ingest: download into raw/ first, then call add_single_file
+    # explicitly so we can clean up the just-downloaded file if it
+    # turns out to be a duplicate (registry already has its hash).
+    # Without this, re-adding the same URL leaves an orphan in raw/
+    # that the registry can't reach via openkb remove.
+    from openkb.url_ingest import looks_like_url, fetch_url_to_raw
+    if looks_like_url(path):
+        fetched = fetch_url_to_raw(path, kb_dir)
+        if fetched is None:
+            return
+        outcome = add_single_file(fetched, kb_dir)
+        # Only clean up on dedup-skip. On "failed" we keep the file so
+        # the user can retry (e.g. transient LLM error during compile)
+        # without re-downloading — and so they don't lose data when
+        # indexing has already succeeded but compilation didn't.
+        if outcome == "skipped":
+            fetched.unlink(missing_ok=True)
+        return
+
     target = Path(path)
     if not target.exists():
         click.echo(f"Path does not exist: {path}")