diff --git a/.agents/specs/hf-model-download.md b/.agents/specs/hf-model-download.md index 7905e61ad..a41b127ca 100644 --- a/.agents/specs/hf-model-download.md +++ b/.agents/specs/hf-model-download.md @@ -246,9 +246,16 @@ Reachability, answered as two separate questions, as [`reachability.md`](../reachability.md) requires. The production entry point is `vllm-server`, which calls `vllm_server_main`, -which parses `--model` at `server_main.cpp:433` and calls the resolver. -`vllm-cli` reaches the same flag. Both are registered command-line paths on -their default configuration. +which parses `--model` and calls the resolver. It is a registered command-line +path on its default configuration. + +`vllm-cli` does NOT reach the resolver, and the earlier sentence claiming it did +was wrong. `examples/cli/main.cpp` includes `vllm.h` and nothing else, because +an example is an application binary interface client only, and this row +deliberately adds no ABI function. So `vllm-cli --model org/repo` and any C ABI +caller of `vllm_model_load` still take a local path. That is listed under +`## Owed`: closing it means adding an ABI entry point, which is a public-surface +decision and its own row. The smallest failing test calls `vllm_server_main(argc, argv)` with `--model org/repo` and `HF_ENDPOINT` aimed at the fake hub, then asserts that @@ -261,7 +268,10 @@ byte for byte: | Mutation | Must turn red | |---|---| -| Delete the resolver call in `server_main.cpp` | The end-to-end case | +| Neutralise the resolver call in `server_main.cpp` with `if (false)` | The end-to-end case | +| Delete the `--revision` branch in `ParseArgs` | The end-to-end case | +| Delete the `--download-dir` branch in `ParseArgs` | The end-to-end case | +| Drop the object-identifier preference in `BlobNameFor` | The blob-name case | | Range mismatch warns instead of failing | The range-ignored case | | Accept an all-identical object identifier listing | The fabricated-identifier case | | Own an identifier with an entry that reports no size | The not-disarmable case | @@ -270,6 +280,17 @@ byte for byte: | Remove index-driven selection | The decoy-file case | | Remove the offline short circuit | The offline cases | +THE RESOLVER MUTATION IS A NEUTRALISATION, NOT A DELETION, and the row above +says `if (false)` for a measured reason. DELETING the call does not compile: +`ResolveModelArgument` is file local, so `server_main.cpp:840` fails with +`error: 'ResolveModelArgument(Args&)' defined but not used +[-Werror=unused-function]` and the exit status is 1. A reviewer who ran the +deletion got a stale binary and a false green, which is exactly the trap this +repository has already recorded. `if (false) ResolveModelArgument(args);` is one +changed line, compiles at exit 0, and reddens the end-to-end case at +`test_serve_hf_model.cpp` `REQUIRE(healthy)` with an empty hub-path list, +because the loader then opens `tiny/llama` as a relative directory. + Each mutation run prints the compiler exit status and `git diff --stat` beside the test result. A mutation that fails to build and a mutation that never applied both read as a passing test, and this tree has recorded both. @@ -308,11 +329,23 @@ asserts a non-zero case count, reads the `Status:` line rather than grepping | W3 | `downloader`: `HEAD`, resume, structural checks, lock, progress | Resume, truncation, and integrity cases green | | W4 | `model_resolver` and the `server_main.cpp` call site | The end-to-end case green, and red when the call site is deleted | | W5 | Build and packaging: the three options, `NOTICE`, `libssl3`, the container check | Every lane builds, and the container check distinguishes a working build from a disabled one | -| W6 | `docs/USAGE.md` and `docs/FEATURES.md` | The new flags, environment variables, and workflow are documented | +| W6 | `docs/guides/hugging-face-access.md`, `docs/reference/server.md` and `docs/FEATURES.md` | The new flags, environment variables, and workflow are documented | | W7 | Fresh review, mutation table, repair | A fresh reviewer returns `PASS` | -W1 through W7 land in one pull request, which is the repository default when no -`## Git integration` preference is recorded. +That plan said W1 through W7 land in one pull request, which is the repository +default when no `## Git integration` preference is recorded. It was overtaken by +what happened. W1 and W2 landed in `31f93787c` through pull request +[#1282](https://github.com/mudler/vllm.cpp/pull/1282), after four fresh reviews +returned findings on the tree-listing size rule, and W3 and W4 land in a SECOND +pull request on `row/ENG-HF-MODEL-DOWNLOAD-W3`. The split is recorded rather +than corrected, because a spec that describes a landing that did not happen is +a false record and this one is read by whoever picks up W5. + +| Stage | Landed in | +|---|---| +| W1, W2 | `31f93787c`, pull request #1282 | +| W3, W4 | `row/ENG-HF-MODEL-DOWNLOAD-W3`, the second pull request | +| W5, W6, W7 | not started | ## Risks and decisions @@ -458,6 +491,89 @@ is recorded here. The regression case for the measured event survives it: the `Lightricks/LTX-2.5` fixture is refused by the degeneracy rule, and its shards are given different sizes so that the size rule would catch it independently. +**A range request answered 200 is REFUSED, where llama.cpp warns.** llama.cpp +`common/download.cpp:222-235 @ b10451` logs "server did not respond with 206" +and carries on. W3 throws. A `200` carries the WHOLE body, and appending a whole +body onto a partial file writes the first N bytes twice. On a safetensors the +data-end rule catches the result, and on a format with no structural proof it is +a silently corrupt weight that a token gate cannot see, because the model still +emits tokens. The refusal costs one re-run and names the partial file to delete. +The refusal is taken in the RESPONSE HANDLER rather than after the transfer, +which is a second decision the same case pinned: with the check after the call, +the suite measured a 12 byte partial file grown to 48 bytes under a refusal that +still reported the right reason. + +**The end-to-end case leaked a listening socket, and the retry loop that hid it +is gone.** This paragraph previously blamed box contention at load 32 and +scheduler starvation, and said that whether a completion can genuinely hang +under contention was not settled by this work. That was WRONG, the cause was +inside this row's own diff, and it is recorded here because a committed spec is +the binding record and a false cause sends the next reader into the serving +core, which this row does not touch. + +`FreePort` in `tests/vllm/entrypoints/openai/test_serve_hf_model.cpp` bound an +ephemeral port with an `httplib::Server` and called `probe.stop()`. That +released nothing. `Server::stop()` at `third_party/httplib/httplib.h:11460` is +guarded by `if (is_running_)`, the function never called `listen_after_bind()`, +so `is_running_` was false and the socket was never shut down, and +`Server::~Server()` is `= default` and does not close it either. httplib sets +`SO_REUSEPORT` at `httplib.h:9455`, so the forked child bound the SAME port +successfully and the kernel load balanced inbound connections between the +orphan, which nobody ever accepted on, and the real server. + +Measured with `ss -ltnp` during an unmutated run at `e25d3d344`: two LISTEN rows +on one port, one owned by the parent and one by the child, the parent's carrying +`Recv-Q 1`, which is the hung request sitting in a backlog nobody drains. + +| Variant | wall, one binary, one box | verdict | +|---|---|---| +| `e25d3d344`, three runs at load 12 to 15 | 90.4 s, 240.4 s, 180.4 s | one FAILED | +| `FreePort` closing its socket, five runs at load 20 to 25 | 0.12 to 0.15 s | five passed | + +Every stall was an exact multiple of a client read timeout, 30 s health and +60 s completion, never an intermediate value. A contended box gives a +distribution, not quantised multiples of the client's own timeout, and the +repaired case is FASTER at load 25 than the committed one was at load 13. + +`FreePort` now uses a POSIX `socket`, `bind`, `getsockname`, `close`, because +the close has to be unconditional. The three-attempt retry loop is DELETED: with +the leak gone there is nothing left for it to absorb, and sixty seconds of dead +time read as a green run with no output at all. The serving core was cleared by +inspection in the same pass, and every wait on the path is a predicate-guarded +`condition_variable::wait`, at `httplib.h:10444`, in +`include/vllm/v1/engine/core_proc.h`, and at +`src/vllm/v1/engine/output_processor.cpp:48`. + +`--max-num-seqs 4` stays on the command line, and the reason recorded for it was +also wrong. The worker-pool arithmetic is true, the default 32 does start a +36-thread pool for one four-token request, but that is not why the case was +slow, and the comment now says so. + +**`--revision` and `--download-dir` were parsed and never reached.** Both +`ParseArgs` branches could be deleted with `test_downloader`, `test_model_resolver` +and `test_serve_hf_model` all still green, because the two cases named for those +flags set `ModelResolveOptions` fields by hand and never touch a flag. That is +`.agents/reachability.md`'s own sentence: a unit test that constructs the type by +hand proves the class works, never that anything reaches it. The end-to-end case +now carries both flags and they are LOAD BEARING. The fake hub publishes the +checkpoint on a non-default branch and lists nothing under the commit `main` +names, so a run without `--revision` fetches nothing and never binds, and the +cache is asserted to land under `--download-dir` and NOT under `HF_HOME`. + +**A blob is named by the object identifier, else by the COMMIT and the path.** +The `## Port map` above says "by the entity tag from the resolve response or by +a locally computed sha256". W3 keeps the object identifier as the first choice +and replaces both fallbacks with the commit and the path, for two reasons. An +entity tag is a transport artifact that a mirror may respell without the bytes +changing, and naming a cache file after one costs a `HEAD` request per file on a +WARM cache, where the correct number of requests is zero and this row has a case +asserting it. A locally computed sha256 has to be computed from the whole file, +and the tree's one sha256 (`kv_cache_utils.h:175`, which the tree may not have a +second of) takes a `std::string`, so a 60 GB shard would be hashed in memory. A +commit plus a repository-relative path already names exactly one byte sequence, +which is the property a content hash was wanted for. The choice the run made is +printed under `--verbose`. + **A null grep is not absence.** This spec states that no document pins a runtime dependency list, based on a grep of `docs/RELEASES.md`, `docs/BUILD.md`, and `scripts/check-release-binary-contract.py`. That grep proves the search terms @@ -465,7 +581,8 @@ wrong, not the fact. W5 confirms by building and running the archive validator. **The GGUF form diverges from vLLM by design.** vLLM has no `org/repo:QUANT` form. This is a tracked exception under the secondary-oracle rule, not a silent -divergence. `docs/USAGE.md` states which upstream defines which form. +divergence. `docs/guides/hugging-face-access.md` states which upstream defines +which form. ## Owed @@ -473,11 +590,37 @@ divergence. `docs/USAGE.md` states which upstream defines which form. - `--tokenizer-revision` and `--code-revision`, `vllm/config/model.py:186,190`. - LoRA adapter fetch, `vllm/lora/utils.py:346`. - The llama.cpp Docker-registry model path, `common/download.cpp:847`. +- **A `.bin`-only repository is fetched and then refused by the loader.** + `SelectWeightFiles` mirrors vLLM's format preference, `["*.safetensors", + "*.bin"]` with the first matching pattern winning + (`default_loader.py:167-184`), so a repository that publishes only + `pytorch_model.bin` is downloaded in full and then refused by `LoadShards` + with `no *.safetensors shards found`. vLLM loads that format and this engine + does not, so mirroring the preference is right and the wasted transfer is + the cost of it. Refusing after phase one, before the weights, would save the + bandwidth and is what the two-phase fetch exists for, but it needs its own + case and its own message. Row `ENG-HF-MODEL-DOWNLOAD`, issue + [#1280](https://github.com/mudler/vllm.cpp/issues/1280). +- **`vllm-cli` and the C ABI do not resolve a repository identifier.** W4 wires + the resolver into `server_main.cpp` only. `examples/cli/main.cpp` is an ABI + client and this row adds no ABI function, so `vllm-cli --model org/repo` and + `vllm_model_load("org/repo")` still take a local path and refuse anything + else exactly as they did. Closing it moves the public surface, so it needs its + own row. Row `ENG-HF-MODEL-DOWNLOAD`, issue + [#1280](https://github.com/mudler/vllm.cpp/issues/1280). - The macOS and Windows TLS lanes, resolved as a table in W5. - The quickstart page, issue [#1281](https://github.com/mudler/vllm.cpp/issues/1281). -- **Wiring `hf_hub` to a production entry point.** W2 lands `hf_hub` reached - only by its own suite. Of `hf_cache`, only `ResolveCachedSnapshotDir` is +- ~~**Wiring `hf_hub` to a production entry point.**~~ CLOSED by W4. The + paragraph below records why the debt existed. `HubResolveCommitCached`, + `HubListRepoFiles`, `HfBlobPath`, `HfSnapshotPath` and + `HfFinalizeSnapshotEntry` are now all reached from + `server_main.cpp`'s `--model` branch through `model_resolver.cpp`, and + `tests/vllm/entrypoints/openai/test_serve_hf_model.cpp` turns red when that + call site is deleted. `HfReadRef` and `HfWriteRef` are reached through + `HubResolveCommitCached` on the same path. + + W2 landed `hf_hub` reached only by its own suite. Of `hf_cache`, only `ResolveCachedSnapshotDir` is reached, through the DFlash draft path in `model_loader.cpp`, and `tests/vllm/entrypoints/test_dflash_draft_hf_cache.cpp` gates that reach by entering the loader with a repository identifier. `HfReadRef`, `HfWriteRef`, @@ -496,7 +639,62 @@ divergence. `docs/USAGE.md` states which upstream defines which form. ## Now -State `READY`, and W1 and W2 have landed. W1 corrected the llama.cpp anchor +State `READY`, and W1 through W4 have landed. W3 added +`src/vllm/transformers_utils/downloader.{h,cpp}`: a `HEAD` probe for size, +entity tag and range support, a `Range: bytes=N-` resume that REFUSES a `200` +answer rather than appending it, a `.incomplete` temporary file renamed only +after the structural proof passes, the safetensors data-end rule and the GGUF +proof through the tree's own reader, a per-repository `flock` beside the +repository directory, progress under `--verbose`, and `SIGINT` cancellation that +keeps the partial file so the next run resumes. + +W4 added `src/vllm/transformers_utils/model_resolver.{h,cpp}` and the call site +in `src/vllm/entrypoints/openai/server_main.cpp`, which is what makes +`--model org/repo` and `--model org/repo:Q4_K_M` fetch and serve. `--revision` +and `--download-dir` are the vLLM flags rather than an invented inline syntax. +The fetch is two phase, configuration JSON before weights, and index driven, so +a published checkpoint's `original/` copy is never requested. + +Three suites cover W3 and W4. `tests/vllm/transformers_utils/test_downloader.cpp` +and `tests/vllm/transformers_utils/test_model_resolver.cpp` run against +in-process fake hubs, and `tests/vllm/entrypoints/openai/test_serve_hf_model.cpp` +is the REACHABILITY gate: it enters at `VllmServerMain(argc, argv)` with +`--model tiny/llama --revision serving --download-dir `, and the server +fetches the committed `llama_embed_e2e` checkpoint from the fake hub, boots, and +completes a `/v1/completions` request. Neutralising the resolver call site in +`server_main.cpp` with `if (false)` leaves the server unable to bind and turns +that case red. DELETING the call instead does not compile, and the mutation +table above records why. + +A FIFTH FRESH REVIEW returned FAIL on six findings, and every one is repaired at +this head. The end-to-end case leaked a listening socket and the retry loop hid +it, both recorded under `## Risks and decisions` with the measurement. +`--revision` and `--download-dir` were parsed and never reached, and the +end-to-end case now carries both and reddens when either `ParseArgs` branch is +deleted. The recorded resolver mutation did not compile, and the table now +records the one that does. The two no-TLS cases were wrapped in +`#ifndef CPPHTTPLIB_OPENSSL_SUPPORT` and would have become zero-assertion skips +the moment W5 defines that macro, so the preprocessor now selects WHICH +statement each case makes rather than whether it makes one. Preprocessing both +files with the macro defined yields zero surviving assertions at `e25d3d344` and +the TLS arm at this head. And the object-identifier preference in `BlobNameFor` +was unpinned, so `test_model_resolver.cpp` now gives one listed file an +identifier and asserts the two blob names apart. + +#1491 landed while this branch was open and split `docs/USAGE.md` into +`docs/guides/`, `docs/models/` and `docs/reference/`. The merge takes that file +unchanged from `origin/main` and re-applies this row's documentation where the +split put it: the fetch, the cache layout and the limits in +`docs/guides/hugging-face-access.md`, and the `--model`, `--revision` and +`--download-dir` rows in `docs/reference/server.md`. + +W5, W6 and W7 have not been done: there is no transport layer security option, +no `docs/FEATURES.md` entry, and no fresh review. + +The paragraph below records the state W2 left, and is kept because the size-rule +findings it names are what the current head's integrity rules are built from. + +W1 and W2 landed first. W1 corrected the llama.cpp anchor table onto stock tag `b10451`. W2 landed `hf_hub` and `hf_cache` under `src/vllm/transformers_utils/`, with `include/vllm/transformers_utils/` headers, and moved the DFlash draft path's copy of the cache walk onto the @@ -526,6 +724,4 @@ documented on the function and on `HfFile::oid`. `test_hf_hub.cpp` now carries The row stays `READY` rather than moving to `ACTIVE`, because an `ACTIVE` row needs a `CLAIM-*` owner recorded in a claim source and that is the operator's -record to write, not an implementer's. W3 through W7 have not been done: there -is no downloader, no `--model` grammar, no transport layer security option, and -no user-facing workflow. `--model` still takes a local path only. +record to write, not an implementer's. diff --git a/CMakeLists.txt b/CMakeLists.txt index ce86ddfac..a73550820 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2347,6 +2347,12 @@ if(VLLM_CPP_SERVER) # vendored cpp-httplib header api_server.cpp uses, so it shares that gate. The # cache half above stays unconditional because it opens no socket. target_sources(vllm PRIVATE src/vllm/transformers_utils/hf_hub.cpp) + # ENG-HF-MODEL-DOWNLOAD W3/W4 (#1280): the byte transport and the `--model` + # grammar. Both speak HTTP through the same vendored header, so both share + # api_server.cpp's gate. The resolver is what `server_main.cpp` calls, so a + # build without the server has no `--model` flag to resolve either. + target_sources(vllm PRIVATE src/vllm/transformers_utils/downloader.cpp) + target_sources(vllm PRIVATE src/vllm/transformers_utils/model_resolver.cpp) target_compile_definitions(vllm PUBLIC VLLM_CPP_SERVER) target_link_libraries(vllm PUBLIC Threads::Threads) # third_party/httplib/httplib.h is reached as (third_party diff --git a/docs/guides/hugging-face-access.md b/docs/guides/hugging-face-access.md index bb554972d..3bbfbabf5 100644 --- a/docs/guides/hugging-face-access.md +++ b/docs/guides/hugging-face-access.md @@ -1,7 +1,78 @@ # Access Hugging Face checkpoints -Pass a local directory or a `.gguf` file to `--model`. The CLI and server do -not download a repository identifier yet. +`--model` takes four forms, and leaves anything else alone. The two local forms +are probed FIRST, so a network call can never shadow a path that exists on +disk. + +| `--model` value | What happens | +|---|---| +| a directory | Opened as it always was. No network | +| a `.gguf` file | Opened as it always was. No network | +| `org/repo` | The checkpoint is fetched into the Hugging Face cache and the snapshot directory is served | +| `org/repo:Q4_K_M` | ONE GGUF file for that quantization is fetched and served | +| anything else | The error it produced before, unchanged | + +`org/repo` mirrors vLLM, which is the only upstream that defines it. +`org/repo:QUANT` is llama.cpp's form and vLLM does not implement it, so it is a +tracked divergence rather than a silent one. + +Two flags go with them, spelled as vLLM spells them. There is deliberately no +inline `org/repo@revision` syntax, because vLLM does not have one. + +| Flag | Meaning | +|---|---| +| `--revision ` | A branch, a tag, or a 40 character commit. Applies to both hub forms | +| `--download-dir ` | The directory that holds the `models--org--repo` folders. Overrides the cache root below | + +This is the SHAPE of a run on a machine that holds no checkpoint. The repository +name is an illustration: no fetch from the live hub has been gated in this tree +yet, because this build speaks no transport layer security and the hermetic +suites run against an in-process fake hub. The online gate is owed by W5 of +[the Hugging Face download specification](../../.agents/specs/hf-model-download.md). + +```sh +vllm-server --model Qwen/Qwen3-0.6B --port 8000 +curl localhost:8000/v1/completions \ + -H 'Content-Type: application/json' \ + -d '{"model":"Qwen/Qwen3-0.6B","prompt":"hello","max_tokens":16}' +``` + +The name a client puts in `"model"` is the name you typed, not the commit +directory the cache happens to hold. + +**`vllm-server` is the only entry point that resolves a repository identifier.** +`vllm-cli` and the C ABI's `vllm_model_load` still take a local path, because an +example is an application binary interface client only and this work adds no ABI +function. + +## How a fetch runs + +The fetch is TWO PHASE. The configuration JSON, the tokenizer and the shard +index come first, and the weights follow, so a repository that is not a model +fails after a few hundred kilobytes rather than after sixty gigabytes. It is +also INDEX DRIVEN: when the repository ships a `model.safetensors.index.json`, +the exact file names in its `weight_map` are fetched, so a published +checkpoint's duplicate-format `original/` directory is never requested. Where +there is no index, `*.safetensors` is preferred over `*.bin` and only the first +of the two that matches is fetched. + +A transfer resumes. A partial file is kept as `{blob}.incomplete` and the next +run asks for `Range: bytes=N-`. **A server that answers that request with `200` +and the whole file is refused, not appended to**: appending a whole body to a +partial file writes the first N bytes twice, and a corrupt weight still emits +tokens, so nothing downstream would notice. Delete the named `.incomplete` file +and run again to start over. `Ctrl-C` cancels and keeps the partial file. + +Nothing takes the name of a finished file until it has proven itself from its +own bytes: a safetensors satisfies `8 + header_len + max(data_offsets[1]) == +file_size`, and a GGUF is opened by this project's own reader, which validates +every tensor span against the real file size. A `Content-Length` that matches is +not accepted as proof on its own. + +`--verbose` prints one line per file, and one lock is taken per repository so +two servers started at once against one cache do not write one blob twice. + +## Environment The library reads these Hugging Face environment variables when it resolves a cache entry: @@ -14,7 +85,9 @@ cache entry: | `HF_HUB_OFFLINE` | Resolve files from the cache without opening a network connection | | `HF_HUB_CACHE`, `HUGGINGFACE_HUB_CACHE`, `HF_HOME`, `XDG_CACHE_HOME`, `HOME` | Cache root, in priority order; `HF_HOME` contributes `$HF_HOME/hub` | -The resolver reads the standard Hugging Face cache layout: +## Cache layout + +The resolver reads and writes the standard Hugging Face cache layout: ```text {hub}/models--org--repo/ @@ -24,19 +97,38 @@ The resolver reads the standard Hugging Face cache layout: ``` If the repository has multiple cached snapshots, the resolver selects the most -recently written snapshot. +recently written snapshot. A cache another tool already populated is read rather +than re-downloaded. + +Where the file system holds no symbolic link, which is the case for a CIFS mount +and can be the case for a `/cache` container volume, a snapshot entry becomes a +real file, and the switch is logged one time for each cache directory it happens +in. + +A cached blob is named by the object identifier the listing carried, and by the +commit plus the repository-relative path when it carried none. `--verbose` says +which of the two the run used. + +A repository listing is refused, rather than partly used, when it fails either +of two integrity checks. An object identifier given to two entries that disagree +on size is refused, because no content hash names two sizes. An identifier whose +characters are all one repeated character is refused, because no content hash +produces one, and an unauthenticated listing for a gated repository is the shape +that produces it. An entry that reports no size can never own an identifier, so +a mirror named by `HF_ENDPOINT` cannot switch the check off by omitting one +field. ## Current limitations -- The CLI and server do not fetch checkpoints. Setting `HF_TOKEN` does not - change a server request until repository downloads reach that entry point. +- This build speaks plain hypertext transfer protocol only, so an `https` + endpoint is refused with a message naming the build options that would add + transport layer security rather than with a connection error. Point + `HF_ENDPOINT` at an `http` mirror until that lands. +- `vllm-cli` and the C ABI do not resolve a repository identifier. Both still + take a local path. - The DFlash draft resolver reads `$HOME/.cache/huggingface/hub`. It does not honor `HF_HOME`. -- Cache-writing support has no public caller yet. - -The cache writer supports filesystems without symbolic links, including CIFS -mounts and some container volumes. On such filesystems it writes a regular file -for each snapshot entry. -See the [Hugging Face download specification](../../.agents/specs/hf-model-download.md) -for implementation state, integrity checks, and remaining work. +All three are recorded under `## Owed` in +[the Hugging Face download specification](../../.agents/specs/hf-model-download.md), +along with the integrity checks and the remaining work. diff --git a/docs/reference/server.md b/docs/reference/server.md index f4b299c7a..39e82d3bd 100644 --- a/docs/reference/server.md +++ b/docs/reference/server.md @@ -140,10 +140,12 @@ a stop token early. | Flag | Default | Meaning | |---|---|---| -| `--model ` | Required except when `--speech-model` selects a speech/music-only server | Model directory (safetensors or `.gguf`) | +| `--model ` | Required except when `--speech-model` selects a speech/music-only server | A local directory or `.gguf` file, opened as before, or a Hugging Face repository, which is fetched into the cache. The local forms are probed first, so a network call can never shadow a path on disk. See [Access Hugging Face checkpoints](../guides/hugging-face-access.md) | +| `--revision ` | repository default branch | A branch, a tag, or a 40 character commit for a `--model org/repo`. vLLM's own flag, and there is no inline `org/repo@rev` syntax | +| `--download-dir ` | the resolved Hugging Face cache root | The directory that holds the `models--org--repo` folders. vLLM's own flag | | `--host H` | `0.0.0.0` | Bind host | | `--port P` | `8000` | Bind port | -| `--served-model-name N` | model dir basename | Model id in `/v1/models` and responses | +| `--served-model-name N` | model dir basename, or the `org/repo` you typed | Model id in `/v1/models` and responses | | `--tokenizer-config F` | `/tokenizer_config.json` | Chat template / tokenizer config | | `--block-size N` | `32` | KV block size. **Must be a multiple of 16**, the attention backends' `get_kv_cache_shape` refuses anything else, and the server now rejects it at startup rather than throwing during engine init | | `--num-blocks N` | `0` (auto, resolves to `256`) | KV block count, and vLLM's `num_gpu_blocks_override`. It wins over every other sizing knob. `0` means auto, which uses `--kv-cache-memory` when that is set and otherwise falls back to `256` blocks | diff --git a/include/vllm/transformers_utils/downloader.h b/include/vllm/transformers_utils/downloader.h new file mode 100644 index 000000000..355046d07 --- /dev/null +++ b/include/vllm/transformers_utils/downloader.h @@ -0,0 +1,163 @@ +// vllm.cpp original: byte transport for a HuggingFace file. Probe, resume, +// structural proof, cross-process lock and progress. +// +// vLLM reaches the bytes through `huggingface_hub`, so the wire behavior has no +// vLLM source to mirror and the structural reference is the secondary oracle +// llama.cpp at stock tag `b10451` (commit +// `10bf611e533d81f739128304991c5e133c6aebd8`): +// - size, entity tag and range probe, `common/download.cpp:321-349` +// - range resume, `common/download.cpp:222-235` +// The cross-process lock mirrors vLLM `weight_utils.py:506` at pin +// `5559679229`, which takes one lock per model before it downloads. +// +// ONE BEHAVIOR DELIBERATELY DIVERGES FROM llama.cpp, and it is the reason this +// file is not a port of that one. See `HubDownloadFile`. +// +// ENG-HF-MODEL-DOWNLOAD W3, issue #1280. +#pragma once + +#include +#include +#include +#include + +#include "vllm/transformers_utils/hf_hub.h" + +namespace vllm { +namespace transformers_utils { + +// What a `HEAD` probe learned about one remote file. Mirrors llama.cpp +// `common/download.cpp:321-349 @ b10451`, which asks the same three questions. +struct HfRemoteFile { + // `Content-Length`, and EMPTY when the answer carried none. It is an optional + // for the reason `HfFile::size` is one: a zero-byte file and an unanswered + // size are different facts, and `0` spells both. Every caller that sizes a + // byte range, a resume offset or a completeness check from it has to be able + // to tell them apart. + std::optional size; + // The entity tag with its quotes and any `W/` prefix removed, or empty. + std::string etag; + // `Accept-Ranges: bytes`. A server that does not say so is never asked to + // resume, because a range it ignores is the failure mode below. + bool accepts_ranges = false; +}; + +// `HEAD {url}`, following a redirect to the content delivery network. The +// bearer token is sent to the host the caller named and to NO redirect target, +// for the reason `hf_hub.cpp`'s client does not set `set_follow_location`. +// +// Throws std::runtime_error on a refusal, and refuses before opening a socket +// when `opts.offline` is set. +HfRemoteFile HubProbeFile(const std::string& url, const HfHubOptions& opts); + +// The three file shapes this row can prove complete STRUCTURALLY, which is the +// only proof it accepts. An opaque remote field is not one: a hub that is not +// answering the truth about a repository can answer a length field too. +enum class HfFileShape { + // Nothing about the interior is known. Only the byte count is checked. + kOpaque, + // `8 + header_len + max(data_offsets[1]) == file_size`. + kSafetensors, + // Magic, version, tensor count, and every tensor span inside the data end. + kGguf, +}; + +// The shape a repository-relative path declares by its extension. +HfFileShape HfShapeForPath(const std::string& path); + +// Prove `file` complete for `shape`. Throws std::runtime_error naming the file +// and the part that did not add up. `kOpaque` returns without reading a byte. +void HfVerifyFileShape(const std::filesystem::path& file, HfFileShape shape); + +struct HfDownloadOptions { + HfHubOptions hub; + // Progress to standard error. The server spells this `--verbose`. + bool verbose = false; +}; + +// What one `HubDownloadFile` call did. `bytes_written` counts what THIS call +// transferred, so a resumed transfer reports the tail and a cache hit reports +// zero. That distinction is the only honest way for a test to state either. +struct HfDownloadResult { + uint64_t bytes_written = 0; + uint64_t file_size = 0; + bool resumed = false; + bool already_present = false; + std::string etag; +}; + +// Fetch `url` into `dest`. +// +// The bytes land in `{dest}.incomplete` and the file is renamed to `dest` only +// after `HfVerifyFileShape` passes, so a partial or malformed transfer never +// occupies the name a later run reads as a cache hit. +// +// An existing `{dest}.incomplete` is RESUMED with `Range: bytes=N-`. +// +// THE DIVERGENCE. llama.cpp warns and continues when a range request is +// answered `200` instead of `206` (`common/download.cpp:222-235 @ b10451`, +// "server did not respond with 206 ... restarting download"; the fork's +// variant logs and keeps the handle). We THROW. A `200` carries the WHOLE body, +// and appending a whole body onto a partial file produces a file whose bytes +// are the first N twice. That file has the wrong length, so the structural +// check catches it here, but the same shape on a format with no structural +// proof is a silently corrupt weight, and a token gate cannot see a corrupt +// weight: it still emits tokens. A refusal that names the cause costs one +// re-run. A corrupt shard costs a debugging session that starts at the model. +// +// `expected_size` is the size the TREE LISTING reported, and it is an optional +// because the listing may report none. It is checked against the transferred +// byte count when it has a value and is not invented when it does not. +// +// Cancels on `SIGINT` once `HfInstallDownloadInterruptHandler` has been called: +// the partial file is left in place, so the next run resumes rather than +// starting again. +// +// Throws std::runtime_error on any refusal, and opens no socket under +// `opts.hub.offline`. +HfDownloadResult HubDownloadFile(const std::string& url, + const std::filesystem::path& dest, + const std::optional& expected_size, + HfFileShape shape, + const HfDownloadOptions& opts); + +// One lock per REPOSITORY, held across processes, mirroring vLLM +// `weight_utils.py:506`. Two `vllm-server` processes started at once against +// one cache must not write one blob twice. +// +// The lock file sits BESIDE the repository directory rather than inside it, so +// it is never mistaken for a snapshot entry and never has to be filtered out of +// the cache walk. An empty `repo_path` takes no lock, because a host with no +// cache directory has nothing to serialize on. +class HfRepoLock { + public: + explicit HfRepoLock(const std::filesystem::path& repo_path); + ~HfRepoLock(); + HfRepoLock(const HfRepoLock&) = delete; + HfRepoLock& operator=(const HfRepoLock&) = delete; + + // True when this object holds a file lock. False when the host has no cache + // directory, or when the platform has no advisory lock: the download still + // runs, because refusing to fetch a model because a lock could not be taken + // is worse than the race the lock prevents. + bool held() const { return fd_ >= 0; } + const std::filesystem::path& path() const { return path_; } + + private: + int fd_ = -1; + std::filesystem::path path_; +}; + +// Install the `SIGINT` handler that cancels an in-flight transfer. Idempotent. +// The previous handler is remembered and re-raised, so `Ctrl-C` still ends the +// process rather than only ending the download. +void HfInstallDownloadInterruptHandler(); + +// True once `SIGINT` arrived after the handler was installed. +bool HfDownloadInterrupted(); + +// Clear the flag. For tests, and for a caller that handled the cancellation. +void HfResetDownloadInterrupt(); + +} // namespace transformers_utils +} // namespace vllm diff --git a/include/vllm/transformers_utils/hf_hub.h b/include/vllm/transformers_utils/hf_hub.h index dd5992347..ae67d3e8c 100644 --- a/include/vllm/transformers_utils/hf_hub.h +++ b/include/vllm/transformers_utils/hf_hub.h @@ -153,6 +153,34 @@ std::string HubResolveCommitCached(const std::string& repo_id, const std::string& revision, const HfHubOptions& opts); +// A parsed hypertext address, and the two checks every call in this file and +// in `downloader.h` runs before it opens a socket. They are declared HERE so +// the byte transport speaks ONE address grammar with the protocol half rather +// than growing a second one that drifts. +// +// Mirrors llama.cpp `common/http.h:33-98 @ b10451`, narrowed to what a hub +// address needs: no user information, because a hub endpoint carries none. +struct HfParsedUrl { + std::string scheme; + std::string host; + int port = 0; + std::string path; // always begins with '/' +}; + +// Throws std::runtime_error naming `url` on a missing scheme, an unterminated +// bracketed IPv6 authority, an unsupported scheme, or an empty host. +HfParsedUrl HfParseUrl(const std::string& url); + +// `[host]` for an IPv6 literal, `host` otherwise. What httplib's scheme-host-port +// constructor needs. +std::string HfFormatHost(const std::string& host); + +// Throw when `url` is `https` and this build carries no transport layer +// security, with a message that NAMES the three build options. A build where +// the option resolved OFF otherwise fails with a connection error that reads +// like a network fault. +void HfRefuseHttpsWithoutTls(const std::string& url); + // True when `repo_id` has the shape the hub accepts: base characters // `[A-Za-z0-9_]`, the special characters `/.-` only between base characters, // and exactly one '/'. Mirrors llama.cpp `common/hf-cache.cpp:121-142`. diff --git a/include/vllm/transformers_utils/model_resolver.h b/include/vllm/transformers_utils/model_resolver.h new file mode 100644 index 000000000..d0575a98f --- /dev/null +++ b/include/vllm/transformers_utils/model_resolver.h @@ -0,0 +1,121 @@ +// vllm.cpp original: the `--model` grammar, and the one entry point behind it. +// +// vLLM is the primary oracle and defines the `org/repo` form at pin +// `5559679229`: +// - local-or-remote decision, `weight_utils.py:345` +// - two-phase fetch, config JSON before weights, `weight_utils.py:349-357` +// - index-driven file selection, `weight_utils.py:472-490` +// - first matching pattern wins, `weight_utils.py:493-496` +// - weight format preference, `default_loader.py:167-184` +// - cross-process download lock, `weight_utils.py:506` +// - `--revision` and `--download-dir` as their own flags, +// `arg_utils.py:839`, `config/model.py:183` +// +// The `org/repo:QUANT` form is the one thing vLLM does not implement, and it +// comes from the secondary oracle llama.cpp at stock tag `b10451`: +// - repository and tag split, `common/download.h:39-42` +// +// ENG-HF-MODEL-DOWNLOAD W4, issue #1280. +#pragma once + +#include +#include +#include + +namespace vllm { +namespace transformers_utils { + +// What the `--model` value turned out to be. `first match wins`, and the two +// local shapes are probed BEFORE anything opens a socket, so a network call can +// never shadow a path that exists on disk. +enum class ModelReference { + // An existing directory. Handed back unchanged. + kLocalDirectory, + // An existing `.gguf` file. Handed back unchanged. + kLocalGgufFile, + // `org/repo`. A vLLM-shaped snapshot: config JSON first, then weights. + kHubSnapshot, + // `org/repo:Q4_K_M`. One GGUF file, llama.cpp's form. + kHubGgufFile, + // Anything else. Handed back unchanged so the existing error still fires. + kUnrecognized, +}; + +// The parsed form of a `--model` value, decided WITHOUT a network call. +struct ParsedModelReference { + ModelReference kind = ModelReference::kUnrecognized; + std::string repo_id; // empty unless the kind is a hub form + std::string tag; // the quantization tag, `kHubGgufFile` only +}; + +// Decide which of the five shapes `model` is. Opens no socket and reads only +// the local file system. +// +// The tag is split on the LAST colon, as llama.cpp does at +// `common/download.h:39-42 @ b10451`. A Windows path such as `C:\models\qwen` +// therefore splits into `C` and `\models\qwen`, and `C` is not a repository +// identifier, so the value falls through to `kUnrecognized` and keeps today's +// behavior. Splitting on the FIRST colon would have made the same path look +// like a repository with a tag. +ParsedModelReference ParseModelReference(const std::string& model); + +struct ModelResolveOptions { + // vLLM's own `--revision`. Applies to both hub forms. Empty means the + // repository's default branch. + std::string revision; + // vLLM's own `--download-dir`. It IS the directory that holds the + // `models--org--repo` folders, which is how `snapshot_download(cache_dir=...)` + // reads it. Empty means the value `HfHubCacheDir()` resolves. + std::filesystem::path download_dir; + // Progress to standard error. + bool verbose = false; +}; + +// Resolve a `--model` value to a local path the loader can open, fetching the +// checkpoint when the value names a repository. +// +// A `kHubSnapshot` returns the snapshot directory. A `kHubGgufFile` returns the +// one `.gguf` file. Every other kind returns `model` unchanged. +// +// TWO vLLM BEHAVIORS ARE MIRRORED RATHER THAN SIMPLIFIED, because skipping +// either costs real bandwidth on this project's checkpoints: +// +// 1. TWO PHASES. The configuration JSON is fetched first and the weights after +// it (`weight_utils.py:349-357`), so a repository that is not a model at +// all fails after a few hundred kilobytes instead of after sixty +// gigabytes. +// 2. INDEX-DRIVEN SELECTION. When the repository holds a +// `model.safetensors.index.json`, the exact names in its `weight_map` are +// fetched (`weight_utils.py:472-490`) rather than everything that matches +// `*.safetensors`. Without it the fetch also pulls duplicate-format +// subdirectories such as `original/`, which on a published checkpoint is a +// second complete copy of the weights. +// +// Format preference falls out of the same mechanism: the patterns +// `["*.safetensors", "*.bin"]` are tried in order and the FIRST one that +// matches the remote listing wins (`default_loader.py:167-184`, +// `weight_utils.py:493-496`). +// +// Throws std::runtime_error on any refusal, with a message that names the +// missing part. +std::string ResolveModelPath(const std::string& model, + const ModelResolveOptions& opts); + +// The file selection, exposed so it can be gated on a listing without a socket. +// `paths` are the repository-relative paths a tree listing reported, and +// `index_weight_map` holds the values of `model.safetensors.index.json`'s +// `weight_map`, or is empty when the repository has no index. +// +// Returns the weight files to fetch, in listing order. +std::vector SelectWeightFiles( + const std::vector& paths, + const std::vector& index_weight_map); + +// True when `path` is a file at the ROOT of the repository, which is the only +// place either phase looks. vLLM lists the root non-recursively +// (`weight_utils.py:349`), so a duplicate-format subdirectory never reaches its +// pattern matching either. +bool IsRepoRootFile(const std::string& path); + +} // namespace transformers_utils +} // namespace vllm diff --git a/src/vllm/entrypoints/openai/server_main.cpp b/src/vllm/entrypoints/openai/server_main.cpp index 96b07104d..8e9a5814e 100644 --- a/src/vllm/entrypoints/openai/server_main.cpp +++ b/src/vllm/entrypoints/openai/server_main.cpp @@ -69,6 +69,7 @@ #include "vllm/entrypoints/chat_template.h" #include "vllm/config/offload.h" #include "vllm/entrypoints/model_loader.h" +#include "vllm/transformers_utils/model_resolver.h" #include #include "vllm/entrypoints/openai/server_main.h" #include "vllm/entrypoints/openai/api_server.h" @@ -170,6 +171,18 @@ int RunFfmpegArgv(const std::vector& args) { struct Args { std::string model_dir; + // ENG-HF-MODEL-DOWNLOAD W4 (#1280): what the user TYPED after `--model`, + // kept because the resolver replaces `model_dir` with a cache path and the + // served model name still defaults to the name the user asked for, exactly + // as vLLM's `served_model_name` defaults to `model` (`arg_utils.py:839`). + std::string model_argument; + // vLLM's own `--revision` (`arg_utils.py:839`, `config/model.py:183`). + // Applies to both HuggingFace forms. There is deliberately no inline + // `org/repo@rev` syntax, because vLLM does not spell it that way. + std::string revision; + // vLLM's own `--download-dir` (`config/model.py:183`): the directory that + // holds the `models--org--repo` folders. + std::string download_dir; std::string host = "0.0.0.0"; int port = 8000; std::string tokenizer_config; // default: /tokenizer_config.json @@ -399,6 +412,7 @@ const InertArg* FindAcceptedInertArg(const std::string& flag) { " [--enable-force-include-usage]\n" " [--enable-tokenizer-info-endpoint]\n" " [--enable-server-dev-mode]\n" + " [--revision REF] [--download-dir DIR]\n" " [--verbose]\n" " [--enable-thinking|--no-enable-thinking]\n" " [--enable-log-requests|--disable-log-requests]\n" @@ -440,6 +454,11 @@ Args ParseArgs(int argc, char** argv) { const std::string flag = argv[i]; if (flag == "--model") { a.model_dir = NextArg(argc, argv, i, argv[0]); + a.model_argument = a.model_dir; + } else if (flag == "--revision") { + a.revision = NextArg(argc, argv, i, argv[0]); + } else if (flag == "--download-dir") { + a.download_dir = NextArg(argc, argv, i, argv[0]); } else if (flag == "--host") { a.host = NextArg(argc, argv, i, argv[0]); } else if (flag == "--port") { @@ -799,6 +818,43 @@ Args ParseArgs(int argc, char** argv) { } +// ENG-HF-MODEL-DOWNLOAD W4 (#1280): turn what the user typed after `--model` +// into a path the loader can open, fetching the checkpoint when it names a +// HuggingFace repository. +// +// THIS IS THE PRODUCTION CALL SITE, and the ONLY one. `vllm-server` reaches +// `--model` through `VllmServerMain`, so a repository identifier that is not +// resolved here is not resolved anywhere. Deleting this call leaves the loader +// opening `org/repo` as a relative path, which is the behavior this row +// replaces, and the end-to-end case in +// `tests/vllm/entrypoints/openai/test_serve_hf_model.cpp` turns red. +// +// `vllm-cli` deliberately does NOT reach it. That example includes `vllm.h` and +// nothing else, because an example is an application binary interface client +// only, and this row adds no ABI function, so `vllm-cli --model org/repo` still +// takes a local path. It is recorded under `## Owed` in +// `.agents/specs/hf-model-download.md`. +// +// An existing directory and an existing `.gguf` file come back unchanged and +// open no socket, so a local run is byte-identical to the one before this row. +void ResolveModelArgument(Args& a) { + if (a.model_dir.empty()) return; + vllm::transformers_utils::ModelResolveOptions opts; + opts.revision = a.revision; + opts.download_dir = a.download_dir; + opts.verbose = a.verbose; + const std::string resolved = + vllm::transformers_utils::ResolveModelPath(a.model_dir, opts); + if (resolved == a.model_dir) return; + // The SERVED name stays the name the user asked for. Without this the + // default would become the 40 character commit directory the cache happens + // to hold, and no client could name the model it just started. + if (a.served_model_name.empty()) a.served_model_name = a.model_argument; + std::cerr << "server: --model " << a.model_argument << " resolved to " + << resolved << "\n"; + a.model_dir = resolved; +} + } // namespace namespace vllm { @@ -807,11 +863,12 @@ namespace openai { int VllmServerMain(int argc, char** argv) { try { - const Args args = ParseArgs(argc, argv); + Args args = ParseArgs(argc, argv); if (args.verbose) { SetEnvironment("VT_SERVER_VERBOSE", "1"); std::cerr << "server: verbose stage logging enabled (debug_stages)\n"; } + ResolveModelArgument(args); { vllm::entrypoints::openai::RequestLogConfig log_cfg; log_cfg.enable_log_requests = args.enable_log_requests; diff --git a/src/vllm/transformers_utils/downloader.cpp b/src/vllm/transformers_utils/downloader.cpp new file mode 100644 index 000000000..7fbab6509 --- /dev/null +++ b/src/vllm/transformers_utils/downloader.cpp @@ -0,0 +1,628 @@ +// See include/vllm/transformers_utils/downloader.h for the llama.cpp `b10451` +// anchors this mirrors and for the ONE behavior it deliberately does not port. +#include "vllm/transformers_utils/downloader.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#if !defined(_WIN32) +#include +#include +#include +#endif + +#include +#include + +#include "vllm/model_executor/model_loader/gguf_reader.h" + +namespace vllm { +namespace transformers_utils { + +namespace fs = std::filesystem; +using nlohmann::json; + +namespace { + +// A transfer follows at most this many redirects. HuggingFace answers a +// `resolve` address with one hop to the content delivery network; the budget +// exists so a mirror that loops cannot spin forever. +constexpr int kMaxRedirects = 5; + +// Progress is printed at most this often, in transferred bytes. A line per +// chunk would drown the terminal on a 60 GB shard. +constexpr uint64_t kProgressStrideBytes = 8ull * 1024 * 1024; + +// `SIGINT` cancellation. `volatile sig_atomic_t` is the only thing a handler may +// touch, so the flag is that and nothing else; every decision the flag drives is +// taken by the transfer loop, outside the handler. +volatile std::sig_atomic_t g_interrupted = 0; +bool g_handler_installed = false; +void (*g_previous_handler)(int) = nullptr; + +extern "C" void HfDownloadSignalHandler(int signal_number) { + g_interrupted = 1; + // Chain to whatever was installed before, so `Ctrl-C` still ends the process + // rather than only ending the download. A handler that swallowed the signal + // would leave the operator pressing it a second time to no effect. + if (g_previous_handler != nullptr && g_previous_handler != SIG_DFL && + g_previous_handler != SIG_IGN) { + g_previous_handler(signal_number); + } +} + +std::string HeaderOrEmpty(const httplib::Response& res, const char* name) { + return res.has_header(name) ? res.get_header_value(name) : std::string(); +} + +// `"abc"` and `W/"abc"` both name the entity `abc`. The quotes and the weakness +// marker are transport spelling, and a blob file must not be named after them. +std::string NormalizeEtag(std::string etag) { + if (etag.rfind("W/", 0) == 0) etag = etag.substr(2); + if (etag.size() >= 2 && etag.front() == '"' && etag.back() == '"') { + etag = etag.substr(1, etag.size() - 2); + } + return etag; +} + +std::optional ParseLength(const std::string& text) { + if (text.empty()) return std::nullopt; + for (const char c : text) { + if (c < '0' || c > '9') return std::nullopt; + } + try { + return static_cast(std::stoull(text)); + } catch (const std::exception&) { + return std::nullopt; + } +} + +// `bytes 100-4095/4096` -> first byte 100, total 4096. Either field is empty +// when the header did not carry it or did not parse. +struct ContentRange { + std::optional first; + std::optional total; +}; + +ContentRange ParseContentRange(const std::string& value) { + ContentRange out; + const size_t space = value.find(' '); + if (space == std::string::npos) return out; + const std::string spec = value.substr(space + 1); + const size_t dash = spec.find('-'); + const size_t slash = spec.find('/'); + if (dash == std::string::npos || slash == std::string::npos || dash > slash) { + return out; + } + out.first = ParseLength(spec.substr(0, dash)); + const std::string total = spec.substr(slash + 1); + if (total != "*") out.total = ParseLength(total); + return out; +} + +// One client aimed at `url`'s authority, with the offline and no-TLS refusals +// already applied. `path` receives the request target that goes with it. +httplib::Client MakeClient(const std::string& url, const HfHubOptions& opts, + std::string* path) { + HfRefuseHttpsWithoutTls(url); + const HfParsedUrl parts = HfParseUrl(url); + *path = parts.path; + httplib::Client client(parts.scheme + "://" + HfFormatHost(parts.host) + ":" + + std::to_string(parts.port)); + // NO `set_follow_location`. httplib copies the whole request, headers + // included, when it follows a redirect, so following one here would hand the + // bearer token to whatever host the answer names. The hops are taken by hand + // below and the token is dropped at the first of them. + client.set_connection_timeout(opts.connect_timeout_seconds, 0); + client.set_read_timeout(opts.read_timeout_seconds, 0); + return client; +} + +httplib::Headers BaseHeaders(const HfHubOptions& opts, bool send_token) { + httplib::Headers headers = {{"User-Agent", "vllm.cpp"}}; + if (send_token && !opts.token.empty()) { + headers.emplace("Authorization", "Bearer " + opts.token); + } + return headers; +} + +bool IsRedirect(int status) { + return status == 301 || status == 302 || status == 303 || status == 307 || + status == 308; +} + +void RefuseOffline(const std::string& url, const HfHubOptions& opts) { + if (!opts.offline) return; + throw std::runtime_error( + "vllm.cpp: HF_HUB_OFFLINE is set, so " + url + + " cannot be fetched. Unset HF_HUB_OFFLINE, or point HF_HOME at a cache " + "that already holds the file."); +} + +void RefuseStatus(int status, const std::string& url) { + if (status == 401 || status == 403) { + throw std::runtime_error( + "vllm.cpp: HuggingFace refused " + url + " with HTTP " + + std::to_string(status) + + ". The repository is private or gated. Set HF_TOKEN (or HF_TOKEN_PATH) " + "to a token that has been granted access to it."); + } + if (status == 404) { + throw std::runtime_error("vllm.cpp: HuggingFace answered HTTP 404 for " + + url + + ". The repository, the revision or the file does " + "not exist."); + } + throw std::runtime_error("vllm.cpp: HuggingFace answered HTTP " + + std::to_string(status) + " for " + url); +} + +std::string HumanBytes(uint64_t bytes) { + static const char* kUnits[] = {"B", "KiB", "MiB", "GiB", "TiB"}; + double value = static_cast(bytes); + int unit = 0; + while (value >= 1024.0 && unit < 4) { + value /= 1024.0; + unit += 1; + } + std::ostringstream out; + out << std::fixed << std::setprecision(unit == 0 ? 0 : 1) << value << ' ' + << kUnits[unit]; + return out.str(); +} + +// The safetensors completeness proof, and the whole reason this row does not +// accept a remote length field as evidence: this arithmetic is answered by the +// file itself. +void VerifySafetensors(const fs::path& file) { + std::ifstream in(file, std::ios::binary); + if (!in) { + throw std::runtime_error("vllm.cpp: cannot read " + file.string() + + " to check that it is a complete safetensors file"); + } + std::error_code ec; + const uint64_t file_size = static_cast(fs::file_size(file, ec)); + if (ec) { + throw std::runtime_error("vllm.cpp: cannot size " + file.string() + ": " + + ec.message()); + } + if (file_size < 8) { + throw std::runtime_error("vllm.cpp: " + file.string() + " is " + + std::to_string(file_size) + + " bytes, which is shorter than a safetensors " + "header length field"); + } + unsigned char length_bytes[8] = {0}; + in.read(reinterpret_cast(length_bytes), 8); + uint64_t header_len = 0; + for (int i = 7; i >= 0; --i) { + header_len = (header_len << 8) | static_cast(length_bytes[i]); + } + if (header_len > file_size - 8) { + throw std::runtime_error( + "vllm.cpp: " + file.string() + " declares a " + + std::to_string(header_len) + " byte safetensors header, which does not " + "fit in its " + std::to_string(file_size) + " bytes"); + } + std::string header(static_cast(header_len), '\0'); + in.read(header.data(), static_cast(header_len)); + if (!in) { + throw std::runtime_error("vllm.cpp: " + file.string() + + " ends inside its safetensors header"); + } + json doc; + try { + doc = json::parse(header); + } catch (const json::exception& e) { + throw std::runtime_error("vllm.cpp: " + file.string() + + " has a safetensors header that is not JSON: " + + e.what()); + } + if (!doc.is_object()) { + throw std::runtime_error("vllm.cpp: " + file.string() + + " has a safetensors header that is not an object"); + } + uint64_t data_end = 0; + for (const auto& [name, entry] : doc.items()) { + if (name == "__metadata__" || !entry.is_object()) continue; + if (!entry.contains("data_offsets") || !entry["data_offsets"].is_array() || + entry["data_offsets"].size() != 2 || + !entry["data_offsets"][1].is_number_unsigned()) { + continue; + } + data_end = std::max(data_end, entry["data_offsets"][1].get()); + } + const uint64_t expected = 8 + header_len + data_end; + if (expected != file_size) { + throw std::runtime_error( + "vllm.cpp: " + file.string() + " is not a complete safetensors file: 8 + " + + std::to_string(header_len) + " header bytes + " + + std::to_string(data_end) + " data bytes is " + + std::to_string(expected) + ", and the file is " + + std::to_string(file_size) + " bytes"); + } +} + +// The GGUF proof runs through `vllm::GgufFile::Open`, which reads the magic and +// the version, caps the counts before it allocates, and validates EVERY tensor +// span against the real file size. That is the data-end check, and it is +// already the tree's one GGUF header reader. A second parser here would be a +// second thing to keep correct. +void VerifyGguf(const fs::path& file) { + try { + const vllm::GgufFile gguf = vllm::GgufFile::Open(file.string()); + if (gguf.Tensors().empty()) { + throw std::runtime_error("it declares no tensor"); + } + } catch (const std::exception& e) { + throw std::runtime_error("vllm.cpp: " + file.string() + + " is not a complete GGUF file: " + e.what()); + } +} + +} // namespace + +void HfInstallDownloadInterruptHandler() { + if (g_handler_installed) return; + g_previous_handler = std::signal(SIGINT, HfDownloadSignalHandler); + g_handler_installed = true; +} + +bool HfDownloadInterrupted() { return g_interrupted != 0; } + +void HfResetDownloadInterrupt() { g_interrupted = 0; } + +HfFileShape HfShapeForPath(const std::string& path) { + const fs::path p(path); + const std::string extension = p.extension().string(); + if (extension == ".safetensors") return HfFileShape::kSafetensors; + if (extension == ".gguf") return HfFileShape::kGguf; + return HfFileShape::kOpaque; +} + +void HfVerifyFileShape(const fs::path& file, HfFileShape shape) { + switch (shape) { + case HfFileShape::kSafetensors: + VerifySafetensors(file); + return; + case HfFileShape::kGguf: + VerifyGguf(file); + return; + case HfFileShape::kOpaque: + return; + } +} + +HfRepoLock::HfRepoLock(const fs::path& repo_path) { + if (repo_path.empty()) return; +#if !defined(_WIN32) + std::error_code ec; + fs::create_directories(repo_path.parent_path(), ec); + path_ = repo_path; + path_ += ".lock"; + fd_ = ::open(path_.c_str(), O_CREAT | O_RDWR | O_CLOEXEC, 0666); + if (fd_ < 0) return; + // BLOCKING, deliberately. vLLM's `weight_utils.py:506` blocks too, and the + // alternative is a second process deciding the model is unavailable when it + // is merely being fetched by the first. + if (::flock(fd_, LOCK_EX) != 0) { + ::close(fd_); + fd_ = -1; + } +#else + // No advisory-lock path on this platform yet. `held()` reports false and the + // download still runs, because refusing to fetch a model because a lock could + // not be taken is worse than the race the lock prevents. + (void)repo_path; +#endif +} + +HfRepoLock::~HfRepoLock() { +#if !defined(_WIN32) + if (fd_ >= 0) { + ::flock(fd_, LOCK_UN); + ::close(fd_); + } +#endif +} + +HfRemoteFile HubProbeFile(const std::string& url, const HfHubOptions& opts) { + RefuseOffline(url, opts); + + HfRemoteFile info; + std::string current = url; + bool send_token = true; + for (int hop = 0; hop <= kMaxRedirects; ++hop) { + std::string path; + httplib::Client client = MakeClient(current, opts, &path); + const httplib::Result res = client.Head(path, BaseHeaders(opts, send_token)); + if (!res) { + throw std::runtime_error("vllm.cpp: cannot reach " + current + ": " + + httplib::to_string(res.error())); + } + // HuggingFace answers the `resolve` address with a redirect and puts the + // content entity tag on THAT answer, so the first tag seen wins and a later + // hop cannot overwrite it with the tag of a storage object. + if (info.etag.empty()) { + std::string etag = HeaderOrEmpty(*res, "X-Linked-Etag"); + if (etag.empty()) etag = HeaderOrEmpty(*res, "ETag"); + info.etag = NormalizeEtag(etag); + } + if (!info.size.has_value()) { + std::string length = HeaderOrEmpty(*res, "X-Linked-Size"); + if (length.empty()) length = HeaderOrEmpty(*res, "Content-Length"); + info.size = ParseLength(length); + } + if (HeaderOrEmpty(*res, "Accept-Ranges") == "bytes") { + info.accepts_ranges = true; + } + if (IsRedirect(res->status)) { + const std::string location = HeaderOrEmpty(*res, "Location"); + if (location.empty()) { + throw std::runtime_error("vllm.cpp: " + current + " answered HTTP " + + std::to_string(res->status) + + " with no Location header"); + } + current = location; + // The credential stops at the host the caller named. + send_token = false; + continue; + } + if (res->status != 200) RefuseStatus(res->status, current); + return info; + } + throw std::runtime_error("vllm.cpp: " + url + " redirected more than " + + std::to_string(kMaxRedirects) + " times"); +} + +HfDownloadResult HubDownloadFile(const std::string& url, const fs::path& dest, + const std::optional& expected_size, + HfFileShape shape, + const HfDownloadOptions& opts) { + HfDownloadResult result; + std::error_code ec; + + // A file already at the destination name is a cache hit ONLY when it proves + // itself. The name was given to it by an earlier run that renamed it after + // the same proof, so this re-check is cheap and it catches an entry a third + // party truncated under the cache. + if (fs::is_regular_file(dest, ec)) { + const uint64_t size = static_cast(fs::file_size(dest, ec)); + if (!ec && (!expected_size.has_value() || *expected_size == size)) { + HfVerifyFileShape(dest, shape); + result.file_size = size; + result.already_present = true; + return result; + } + } + + RefuseOffline(url, opts.hub); + fs::create_directories(dest.parent_path(), ec); + + fs::path temp = dest; + temp += ".incomplete"; + + uint64_t offset = 0; + if (fs::is_regular_file(temp, ec)) { + const uint64_t partial = static_cast(fs::file_size(temp, ec)); + if (!ec) offset = partial; + } + + const HfRemoteFile probe = HubProbeFile(url, opts.hub); + std::optional total = probe.size.has_value() ? probe.size : expected_size; + if (probe.size.has_value() && expected_size.has_value() && + *probe.size != *expected_size) { + throw std::runtime_error( + "vllm.cpp: the tree listing gives " + url + " a size of " + + std::to_string(*expected_size) + " bytes and the file answers " + + std::to_string(*probe.size) + + " bytes. The two disagree, so neither can be used to prove the " + "transfer complete."); + } + + if (offset > 0) { + // A partial file that is already the whole file needs no request, and a + // partial file the server cannot resume is started again rather than + // appended to. + const bool complete = total.has_value() && offset == *total; + if (!complete && (!probe.accepts_ranges || (total.has_value() && offset > *total))) { + fs::remove(temp, ec); + offset = 0; + } + } + + const bool resume = offset > 0; + result.resumed = resume; + result.etag = probe.etag; + + if (!total.has_value() || offset < *total) { + std::ofstream out(temp, std::ios::binary | + (resume ? std::ios::app : std::ios::trunc)); + if (!out) { + throw std::runtime_error("vllm.cpp: cannot open " + temp.string() + + " for writing"); + } + + std::string current = url; + bool send_token = true; + bool done = false; + for (int hop = 0; hop <= kMaxRedirects && !done; ++hop) { + std::string path; + httplib::Client client = MakeClient(current, opts.hub, &path); + httplib::Headers headers = BaseHeaders(opts.hub, send_token); + if (resume) { + headers.emplace("Range", "bytes=" + std::to_string(offset) + "-"); + } + + int status = 0; + std::string location; + std::optional body_length; + ContentRange range; + uint64_t received = 0; + uint64_t next_report = kProgressStrideBytes; + bool cancelled = false; + + const httplib::Result res = client.Get( + path, headers, + [&](const httplib::Response& response) { + status = response.status; + location = HeaderOrEmpty(response, "Location"); + body_length = ParseLength(HeaderOrEmpty(response, "Content-Length")); + range = ParseContentRange(HeaderOrEmpty(response, "Content-Range")); + // A RESUMED transfer accepts ONLY 206, and the refusal is taken + // HERE rather than after the call, because returning true would + // hand the body to the receiver below and the first `offset` bytes + // would already be on disk twice by the time the throw ran. + // Measured: this suite saw a 12 byte partial file grow to 48 bytes + // under a refusal that still reported the right reason. + if (resume) return status == 206; + return status == 200; + }, + [&](const char* data, size_t length) { + if (HfDownloadInterrupted()) { + cancelled = true; + return false; + } + out.write(data, static_cast(length)); + if (!out) return false; + received += length; + if (opts.verbose && received >= next_report) { + next_report = received + kProgressStrideBytes; + std::cerr << "download: " << dest.filename().string() << ' ' + << HumanBytes(offset + received); + if (total.has_value()) { + std::cerr << " / " << HumanBytes(*total); + } + std::cerr << std::endl; + } + return true; + }); + + if (cancelled) { + out.close(); + throw std::runtime_error( + "vllm.cpp: the download of " + url + + " was cancelled by SIGINT. The partial file is kept at " + + temp.string() + ", so the next run resumes rather than starting " + "again."); + } + + if (IsRedirect(status)) { + if (location.empty()) { + throw std::runtime_error("vllm.cpp: " + current + " answered HTTP " + + std::to_string(status) + + " with no Location header"); + } + current = location; + send_token = false; + continue; + } + + // THE DIVERGENCE FROM llama.cpp `common/download.cpp:222-235 @ b10451`, + // which warns and continues here. A 200 answer to a range request carries + // the WHOLE body, and appending a whole body to a partial file writes the + // first `offset` bytes twice. See downloader.h. + if (resume && status == 200) { + out.close(); + throw std::runtime_error( + "vllm.cpp: " + current + " was asked for bytes " + + std::to_string(offset) + + "- and answered HTTP 200 with the whole file instead of HTTP 206. " + "Appending that body to the partial file at " + temp.string() + + " would write the first " + std::to_string(offset) + + " bytes twice and silently corrupt the weight, so the transfer is " + "refused. Delete that file and run again to fetch it from the " + "start."); + } + if (status != 200 && status != 206) { + out.close(); + RefuseStatus(status, current); + } + if (resume && range.first.has_value() && *range.first != offset) { + out.close(); + throw std::runtime_error( + "vllm.cpp: " + current + " was asked for bytes " + + std::to_string(offset) + "- and answered with a range starting at " + + std::to_string(*range.first) + ". The two do not line up, so the " + "transfer is refused rather than written to the wrong offset."); + } + if (range.total.has_value() && !total.has_value()) total = range.total; + + // THE BYTE COUNT COMES FIRST, ahead of the transport's own verdict. A + // body shorter than the length its own answer declared is a TRUNCATED + // transfer, and httplib reports some of these as a read error and + // returns others as a successful short body. With this check placed + // AFTER `!res`, the transport's generic message won every truncation + // this suite could construct, and deleting the check left the truncation + // case GREEN, so the guarantee had no test. Ordered this way the specific + // diagnosis wins and the case measures the check rather than httplib. + if (body_length.has_value() && received != *body_length) { + out.close(); + throw std::runtime_error( + "vllm.cpp: " + current + " declared a body of " + + std::to_string(*body_length) + " bytes and delivered " + + std::to_string(received) + + ". The transfer is truncated and is refused."); + } + if (!res) { + out.close(); + throw std::runtime_error( + "vllm.cpp: the transfer of " + current + " ended after " + + std::to_string(offset + received) + " bytes: " + + httplib::to_string(res.error())); + } + if (!out) { + out.close(); + throw std::runtime_error("vllm.cpp: writing " + temp.string() + + " failed after " + + std::to_string(offset + received) + " bytes"); + } + result.bytes_written = received; + done = true; + } + + out.close(); + if (!done) { + throw std::runtime_error("vllm.cpp: " + url + " redirected more than " + + std::to_string(kMaxRedirects) + " times"); + } + } + + const uint64_t written = static_cast(fs::file_size(temp, ec)); + if (ec) { + throw std::runtime_error("vllm.cpp: cannot size " + temp.string() + ": " + + ec.message()); + } + if (total.has_value() && written != *total) { + throw std::runtime_error( + "vllm.cpp: " + url + " is " + std::to_string(*total) + + " bytes and the transfer left " + std::to_string(written) + + " bytes in " + temp.string() + ". The transfer is refused."); + } + + // The STRUCTURAL proof, before the rename and never after it. The destination + // name is what a later run reads as a cache hit, so a file may only take that + // name once it has proven itself from its own bytes. + HfVerifyFileShape(temp, shape); + + fs::remove(dest, ec); + fs::rename(temp, dest, ec); + if (ec) { + throw std::runtime_error("vllm.cpp: cannot move " + temp.string() + " to " + + dest.string() + ": " + ec.message()); + } + result.file_size = written; + return result; +} + +} // namespace transformers_utils +} // namespace vllm diff --git a/src/vllm/transformers_utils/hf_hub.cpp b/src/vllm/transformers_utils/hf_hub.cpp index 7d75215d5..81f99272f 100644 --- a/src/vllm/transformers_utils/hf_hub.cpp +++ b/src/vllm/transformers_utils/hf_hub.cpp @@ -74,73 +74,6 @@ bool IsSafeEntryPath(const std::string& path) { return true; } -struct ParsedUrl { - std::string scheme; - std::string host; - int port = 0; - std::string path; // always begins with '/' -}; - -// Mirrors llama.cpp `common/http.h:33-98 @ b10451`, narrowed to what an -// endpoint needs: no user information, because a hub endpoint carries none. -ParsedUrl ParseUrl(const std::string& url) { - ParsedUrl parts; - const size_t scheme_end = url.find("://"); - if (scheme_end == std::string::npos) { - throw std::runtime_error("vllm.cpp: HF_ENDPOINT '" + url + - "' has no scheme; expected http:// or https://"); - } - parts.scheme = url.substr(0, scheme_end); - std::string rest = url.substr(scheme_end + 3); - - const size_t slash = rest.find('/'); - if (slash != std::string::npos) { - parts.host = rest.substr(0, slash); - parts.path = rest.substr(slash); - } else { - parts.host = rest; - parts.path = "/"; - } - - std::string port_text; - if (!parts.host.empty() && parts.host.front() == '[') { - const size_t close = parts.host.find(']'); - if (close == std::string::npos) { - throw std::runtime_error("vllm.cpp: HF_ENDPOINT '" + url + - "' has an unterminated IPv6 host"); - } - const std::string after = parts.host.substr(close + 1); - if (!after.empty() && after.front() == ':') port_text = after.substr(1); - parts.host = parts.host.substr(1, close - 1); - } else { - const size_t colon = parts.host.find(':'); - if (colon != std::string::npos) { - port_text = parts.host.substr(colon + 1); - parts.host = parts.host.substr(0, colon); - } - } - - if (!port_text.empty()) { - parts.port = std::stoi(port_text); - } else if (parts.scheme == "http") { - parts.port = 80; - } else if (parts.scheme == "https") { - parts.port = 443; - } else { - throw std::runtime_error("vllm.cpp: HF_ENDPOINT '" + url + - "' uses the unsupported scheme '" + parts.scheme + - "'"); - } - if (parts.host.empty()) { - throw std::runtime_error("vllm.cpp: HF_ENDPOINT '" + url + "' has no host"); - } - return parts; -} - -std::string FormatHost(const std::string& host) { - return host.find(':') != std::string::npos ? "[" + host + "]" : host; -} - // GET a JSON document from the hub. `repo_id` appears in every refusal, because // a message that does not name the repository cannot be acted on. json ApiGet(const HfHubOptions& opts, const std::string& relative_path, @@ -158,21 +91,11 @@ json ApiGet(const HfHubOptions& opts, const std::string& relative_path, "holds the repository."); } - const ParsedUrl url = ParseUrl(opts.endpoint); + const HfParsedUrl url = HfParseUrl(opts.endpoint); -#ifndef CPPHTTPLIB_OPENSSL_SUPPORT - if (url.scheme == "https") { - throw std::runtime_error( - "vllm.cpp: this build cannot speak HTTPS, so it cannot reach " + - opts.endpoint + - ". Rebuild with -DVLLM_CPP_HF_DOWNLOAD=ON and one of " - "-DVLLM_CPP_OPENSSL=ON (default, needs the OpenSSL development files) " - "or -DVLLM_CPP_BUILD_BORINGSSL=ON, or set HF_ENDPOINT to an http:// " - "mirror."); - } -#endif + HfRefuseHttpsWithoutTls(opts.endpoint); - httplib::Client client(url.scheme + "://" + FormatHost(url.host) + ":" + + httplib::Client client(url.scheme + "://" + HfFormatHost(url.host) + ":" + std::to_string(url.port)); // NO `set_follow_location(true)`. httplib copies the whole request, headers // included, when it follows a redirect (third_party/httplib/httplib.h:7774) @@ -254,6 +177,79 @@ std::vector> CollectRefs(const json& doc) { } // namespace +HfParsedUrl HfParseUrl(const std::string& url) { + HfParsedUrl parts; + const size_t scheme_end = url.find("://"); + if (scheme_end == std::string::npos) { + throw std::runtime_error("vllm.cpp: HF_ENDPOINT '" + url + + "' has no scheme; expected http:// or https://"); + } + parts.scheme = url.substr(0, scheme_end); + std::string rest = url.substr(scheme_end + 3); + + const size_t slash = rest.find('/'); + if (slash != std::string::npos) { + parts.host = rest.substr(0, slash); + parts.path = rest.substr(slash); + } else { + parts.host = rest; + parts.path = "/"; + } + + std::string port_text; + if (!parts.host.empty() && parts.host.front() == '[') { + const size_t close = parts.host.find(']'); + if (close == std::string::npos) { + throw std::runtime_error("vllm.cpp: HF_ENDPOINT '" + url + + "' has an unterminated IPv6 host"); + } + const std::string after = parts.host.substr(close + 1); + if (!after.empty() && after.front() == ':') port_text = after.substr(1); + parts.host = parts.host.substr(1, close - 1); + } else { + const size_t colon = parts.host.find(':'); + if (colon != std::string::npos) { + port_text = parts.host.substr(colon + 1); + parts.host = parts.host.substr(0, colon); + } + } + + if (!port_text.empty()) { + parts.port = std::stoi(port_text); + } else if (parts.scheme == "http") { + parts.port = 80; + } else if (parts.scheme == "https") { + parts.port = 443; + } else { + throw std::runtime_error("vllm.cpp: HF_ENDPOINT '" + url + + "' uses the unsupported scheme '" + parts.scheme + + "'"); + } + if (parts.host.empty()) { + throw std::runtime_error("vllm.cpp: HF_ENDPOINT '" + url + "' has no host"); + } + return parts; +} + +std::string HfFormatHost(const std::string& host) { + return host.find(':') != std::string::npos ? "[" + host + "]" : host; +} + +void HfRefuseHttpsWithoutTls(const std::string& url) { +#ifndef CPPHTTPLIB_OPENSSL_SUPPORT + if (url.rfind("https://", 0) == 0) { + throw std::runtime_error( + "vllm.cpp: this build cannot speak HTTPS, so it cannot reach " + url + + ". Rebuild with -DVLLM_CPP_HF_DOWNLOAD=ON and one of " + "-DVLLM_CPP_OPENSSL=ON (default, needs the OpenSSL development files) " + "or -DVLLM_CPP_BUILD_BORINGSSL=ON, or set HF_ENDPOINT to an http:// " + "mirror."); + } +#else + (void)url; +#endif +} + bool IsValidHfRepoId(const std::string& repo_id) { // Mirrors llama.cpp `common/hf-cache.cpp:121-142 @ b10451`: base characters // [A-Za-z0-9_] are always valid, the special characters [/.-] must sit diff --git a/src/vllm/transformers_utils/model_resolver.cpp b/src/vllm/transformers_utils/model_resolver.cpp new file mode 100644 index 000000000..efea079ec --- /dev/null +++ b/src/vllm/transformers_utils/model_resolver.cpp @@ -0,0 +1,423 @@ +// See include/vllm/transformers_utils/model_resolver.h for the grammar this +// implements and for the vLLM `5559679229` anchors it mirrors. +#include "vllm/transformers_utils/model_resolver.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/transformers_utils/downloader.h" +#include "vllm/transformers_utils/hf_cache.h" +#include "vllm/transformers_utils/hf_hub.h" + +namespace vllm { +namespace transformers_utils { + +namespace fs = std::filesystem; +using nlohmann::json; + +namespace { + +// PHASE ONE. Everything a repository needs before anybody looks at a weight: +// the model configuration, the tokenizer, and the shard index. A repository +// that is not a model at all fails here, after a few hundred kilobytes. +// Mirrors vLLM fetching the configuration through `transformers` before +// `download_weights_from_hf` runs (`weight_utils.py:349-357`). +bool IsConfigPhaseFile(const std::string& path) { + const fs::path p(path); + const std::string extension = p.extension().string(); + return extension == ".json" || extension == ".txt" || extension == ".model" || + extension == ".jinja"; +} + +// PHASE TWO, in preference order, first matching pattern wins +// (`default_loader.py:167-184`, `weight_utils.py:493-496`). +const char* const kWeightExtensions[] = {".safetensors", ".bin"}; + +std::string ToLower(std::string text) { + for (char& c : text) { + c = static_cast(std::tolower(static_cast(c))); + } + return text; +} + +// A blob file name that names one file on a case-sensitive file system and on a +// case-insensitive one alike, and that can never escape the blobs directory. +std::string FlattenPath(const std::string& path) { + std::string flat; + flat.reserve(path.size()); + for (const char c : path) { + const bool safe = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || + (c >= '0' && c <= '9') || c == '.' || c == '-' || c == '_'; + flat.push_back(safe ? c : '_'); + } + return flat; +} + +// WHICH NAME A BLOB GETS, and it is decided WITHOUT a request. +// +// The object identifier wins when the listing carried one that survived the +// integrity rules, because it is a content hash and two repositories that hold +// one byte sequence then share one blob. +// +// Otherwise the name is the COMMIT plus the path. The commit is what every call +// after the reference resolution names, so a commit and a repository-relative +// path already identify exactly one byte sequence, which is the property a +// content hash was wanted for. It is deliberately NOT the entity tag: a tag is +// a transport artifact that a mirror may respell without the bytes changing, +// and naming a cache file after one would have cost a `HEAD` request per file +// on a WARM cache, where the correct number of requests is zero. +std::string BlobNameFor(const HfFile& file, const std::string& commit) { + if (!file.oid.empty()) return file.oid; + return commit + "--" + FlattenPath(file.path); +} + +// The quantization tag a GGUF file name carries: `Model-Q4_K_M.gguf` -> `q4_k_m`, +// and `Model-Q4_K_M-00001-of-00003.gguf` -> `q4_k_m`. Empty when the name +// carries none. Mirrors llama.cpp matching a tag against the file names a +// repository holds rather than against a manifest. +std::string GgufTagOf(const std::string& path) { + const fs::path p(path); + if (p.extension().string() != ".gguf") return std::string(); + std::string stem = p.stem().string(); + // Drop a `-00001-of-00003` shard suffix so every shard of one quantization + // reports the same tag. + const size_t of = ToLower(stem).rfind("-of-"); + if (of != std::string::npos) { + const size_t dash = stem.rfind('-', of - 1); + if (dash != std::string::npos) stem = stem.substr(0, dash); + } + const size_t dash = stem.rfind('-'); + if (dash == std::string::npos) return std::string(); + return ToLower(stem.substr(dash + 1)); +} + +std::string ReadFileToString(const fs::path& path) { + std::ifstream in(path, std::ios::binary); + if (!in) return std::string(); + return std::string((std::istreambuf_iterator(in)), + std::istreambuf_iterator()); +} + +// The `weight_map` VALUES of a `model.safetensors.index.json`, de-duplicated in +// first-seen order. Empty when the file is absent or carries no map, which is +// how a single-shard repository reads. +std::vector ReadIndexWeightMap(const fs::path& index_file) { + std::vector names; + std::error_code ec; + if (!fs::is_regular_file(index_file, ec)) return names; + json doc; + try { + doc = json::parse(ReadFileToString(index_file)); + } catch (const json::exception&) { + return names; + } + if (!doc.is_object() || !doc.contains("weight_map") || + !doc["weight_map"].is_object()) { + return names; + } + std::set seen; + for (const auto& [tensor, shard] : doc["weight_map"].items()) { + (void)tensor; + if (!shard.is_string()) continue; + const std::string name = shard.get(); + if (seen.insert(name).second) names.push_back(name); + } + return names; +} + +// Fetch one listed file into the cache and place its snapshot entry. Returns +// the snapshot path. +fs::path FetchOne(const HfFile& file, const std::string& commit, + const fs::path& repo_path, const HfDownloadOptions& dopts) { + const fs::path blob = HfBlobPath(repo_path, BlobNameFor(file, commit)); + const fs::path final_path = HfSnapshotPath(repo_path, commit) / file.path; + const HfFileShape shape = HfShapeForPath(file.path); + + std::error_code ec; + const HfDownloadResult result = + HubDownloadFile(file.url, blob, file.size, shape, dopts); + if (dopts.verbose) { + std::cerr << "download: " << file.path << ' ' + << (result.already_present + ? "already in the cache" + : (result.resumed ? "resumed" : "fetched")) + << ", blob " << blob.filename().string() + << (file.oid.empty() ? " (named by commit and path)" + : " (named by object identifier)") + << std::endl; + } + + fs::create_directories(final_path.parent_path(), ec); + if (!HfFinalizeSnapshotEntry(blob, final_path)) { + throw std::runtime_error("vllm.cpp: cannot place the snapshot entry " + + final_path.string() + " for the cached blob " + + blob.string()); + } + return final_path; +} + +const HfFile* FindByPath(const std::vector& files, + const std::string& path) { + for (const HfFile& file : files) { + if (file.path == path) return &file; + } + return nullptr; +} + +HfHubOptions HubOptionsFor(const ModelResolveOptions& opts) { + HfHubOptions hub = HfHubOptionsFromEnv(); + if (!opts.download_dir.empty()) { + // vLLM's `--download-dir` IS the directory that holds the + // `models--org--repo` folders: it is handed to + // `snapshot_download(cache_dir=...)` (`config/model.py:183`). + hub.hub_dir = opts.download_dir; + } + return hub; +} + +// `org/repo` -> the snapshot directory, fetching what the cache lacks. +std::string ResolveSnapshot(const std::string& repo_id, + const ModelResolveOptions& opts) { + const HfHubOptions hub = HubOptionsFor(opts); + const std::string commit = HubResolveCommitCached(repo_id, opts.revision, hub); + const fs::path repo_path = HfRepoPath(hub.hub_dir, repo_id); + if (repo_path.empty()) { + throw std::runtime_error( + "vllm.cpp: repository '" + repo_id + + "' cannot be fetched because this host has no HuggingFace cache " + "directory. Set HF_HOME, or pass --download-dir."); + } + const fs::path snapshot = HfSnapshotPath(repo_path, commit); + + std::error_code ec; + if (hub.offline) { + if (fs::is_regular_file(snapshot / "config.json", ec)) return snapshot.string(); + throw std::runtime_error( + "vllm.cpp: HF_HUB_OFFLINE is set and the cache under " + + snapshot.string() + + " holds no config.json for repository '" + repo_id + + "'. Fetch it once with HF_HUB_OFFLINE unset, or point HF_HOME at a " + "cache that already holds it."); + } + + // ONE lock per repository, across processes, mirroring vLLM + // `weight_utils.py:506`. Two servers started at once against one cache must + // not write one blob twice. + const HfRepoLock lock(repo_path); + HfInstallDownloadInterruptHandler(); + + const std::vector files = HubListRepoFiles(repo_id, commit, hub); + + HfDownloadOptions dopts; + dopts.hub = hub; + dopts.verbose = opts.verbose; + + // PHASE ONE. + std::vector root_paths; + for (const HfFile& file : files) { + if (!IsRepoRootFile(file.path)) continue; + root_paths.push_back(file.path); + if (IsConfigPhaseFile(file.path)) FetchOne(file, commit, repo_path, dopts); + } + if (!fs::is_regular_file(snapshot / "config.json", ec)) { + throw std::runtime_error( + "vllm.cpp: repository '" + repo_id + "' at revision " + commit + + " has no config.json at its root, so it is not a model checkpoint this " + "server can load. Nothing beyond its configuration was fetched."); + } + + // PHASE TWO. + const std::vector index_weight_map = + ReadIndexWeightMap(snapshot / "model.safetensors.index.json"); + const std::vector weights = + SelectWeightFiles(root_paths, index_weight_map); + if (weights.empty()) { + throw std::runtime_error( + "vllm.cpp: repository '" + repo_id + "' at revision " + commit + + " holds no *.safetensors and no *.bin weight file at its root."); + } + for (const std::string& path : weights) { + const HfFile* file = FindByPath(files, path); + if (file == nullptr) { + throw std::runtime_error( + "vllm.cpp: repository '" + repo_id + + "' has a model.safetensors.index.json whose weight_map names '" + + path + + "', and the tree listing does not hold that file. The index and the " + "listing disagree, so the checkpoint is not usable."); + } + FetchOne(*file, commit, repo_path, dopts); + } + return snapshot.string(); +} + +// `org/repo:Q4_K_M` -> the one GGUF file. +std::string ResolveGgufFile(const std::string& repo_id, const std::string& tag, + const ModelResolveOptions& opts) { + const HfHubOptions hub = HubOptionsFor(opts); + const std::string commit = HubResolveCommitCached(repo_id, opts.revision, hub); + const fs::path repo_path = HfRepoPath(hub.hub_dir, repo_id); + if (repo_path.empty()) { + throw std::runtime_error( + "vllm.cpp: repository '" + repo_id + + "' cannot be fetched because this host has no HuggingFace cache " + "directory. Set HF_HOME, or pass --download-dir."); + } + const std::string wanted = ToLower(tag); + + std::error_code ec; + if (hub.offline) { + const fs::path snapshot = HfSnapshotPath(repo_path, commit); + for (const fs::directory_entry& entry : + fs::directory_iterator(snapshot, ec)) { + if (entry.is_regular_file(ec) && + GgufTagOf(entry.path().filename().string()) == wanted) { + return entry.path().string(); + } + } + throw std::runtime_error( + "vllm.cpp: HF_HUB_OFFLINE is set and the cache under " + + snapshot.string() + " holds no '" + tag + "' GGUF file for repository '" + + repo_id + "'."); + } + + const HfRepoLock lock(repo_path); + HfInstallDownloadInterruptHandler(); + + const std::vector files = HubListRepoFiles(repo_id, commit, hub); + + std::vector matches; + std::set offered; + for (const HfFile& file : files) { + const std::string file_tag = GgufTagOf(file.path); + if (file_tag.empty()) continue; + offered.insert(file_tag); + if (file_tag == wanted) matches.push_back(&file); + } + if (matches.empty()) { + std::string tags; + for (const std::string& name : offered) { + if (!tags.empty()) tags += ", "; + tags += name; + } + throw std::runtime_error( + "vllm.cpp: repository '" + repo_id + "' at revision " + commit + + " holds no GGUF file for the tag '" + tag + "'. It holds " + + (tags.empty() ? std::string("no GGUF file at all") : tags) + "."); + } + + HfDownloadOptions dopts; + dopts.hub = hub; + dopts.verbose = opts.verbose; + + // A quantization split across shards is fetched WHOLE and the FIRST shard is + // returned, because that is the name the GGUF reader opens and it pulls its + // siblings in from the same directory. + std::sort(matches.begin(), matches.end(), + [](const HfFile* a, const HfFile* b) { return a->path < b->path; }); + fs::path first; + for (const HfFile* file : matches) { + const fs::path placed = FetchOne(*file, commit, repo_path, dopts); + if (first.empty()) first = placed; + } + return first.string(); +} + +} // namespace + +bool IsRepoRootFile(const std::string& path) { + return !path.empty() && path.find('/') == std::string::npos; +} + +std::vector SelectWeightFiles( + const std::vector& paths, + const std::vector& index_weight_map) { + // INDEX-DRIVEN SELECTION (`weight_utils.py:472-490`). When the repository + // ships a shard index, the exact names in its `weight_map` are the answer. + // Pattern matching would additionally pull every duplicate-format copy of the + // weights the repository happens to carry, which on a published checkpoint is + // a second complete set of shards. + if (!index_weight_map.empty()) return index_weight_map; + + for (const char* extension : kWeightExtensions) { + std::vector matches; + for (const std::string& path : paths) { + if (!IsRepoRootFile(path)) continue; + if (fs::path(path).extension().string() == extension) { + matches.push_back(path); + } + } + // FIRST MATCHING PATTERN WINS (`weight_utils.py:493-496`). A repository + // that ships both formats is fetched once, in safetensors. + if (!matches.empty()) return matches; + } + return {}; +} + +ParsedModelReference ParseModelReference(const std::string& model) { + ParsedModelReference parsed; + if (model.empty()) return parsed; + + std::error_code ec; + // THE LOCAL PROBES COME FIRST, and that order is the requirement rather than + // an optimization: a network call must never shadow a path that exists on + // disk. + if (fs::is_directory(model, ec)) { + parsed.kind = ModelReference::kLocalDirectory; + return parsed; + } + if (fs::is_regular_file(model, ec) && fs::path(model).extension() == ".gguf") { + parsed.kind = ModelReference::kLocalGgufFile; + return parsed; + } + + // Split on the LAST colon, as llama.cpp does at `common/download.h:39-42 @ + // b10451`. See the header for the Windows path this protects. + const size_t colon = model.rfind(':'); + if (colon != std::string::npos && colon + 1 < model.size()) { + const std::string repo = model.substr(0, colon); + const std::string tag = model.substr(colon + 1); + if (IsValidHfRepoId(repo)) { + parsed.kind = ModelReference::kHubGgufFile; + parsed.repo_id = repo; + parsed.tag = tag; + return parsed; + } + } + + if (IsValidHfRepoId(model)) { + parsed.kind = ModelReference::kHubSnapshot; + parsed.repo_id = model; + return parsed; + } + return parsed; +} + +std::string ResolveModelPath(const std::string& model, + const ModelResolveOptions& opts) { + const ParsedModelReference parsed = ParseModelReference(model); + switch (parsed.kind) { + case ModelReference::kLocalDirectory: + case ModelReference::kLocalGgufFile: + case ModelReference::kUnrecognized: + // Unchanged, and no socket. An unrecognized value reaches the loader + // exactly as it did before this row, so its existing error still fires. + return model; + case ModelReference::kHubSnapshot: + return ResolveSnapshot(parsed.repo_id, opts); + case ModelReference::kHubGgufFile: + return ResolveGgufFile(parsed.repo_id, parsed.tag, opts); + } + return model; +} + +} // namespace transformers_utils +} // namespace vllm diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 8e5816c94..1c0770971 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -1521,6 +1521,12 @@ if(VLLM_CPP_SERVER) # hub. It starts a REAL cpp-httplib server on an ephemeral port, so it shares # api_server.cpp's gate and the RUN_SERIAL treatment two rows below. vllm_cpp_add_test(test_hf_hub vllm/transformers_utils/test_hf_hub.cpp) + # ENG-HF-MODEL-DOWNLOAD W3/W4 (#1280): the byte transport and the `--model` + # grammar, both against an in-process fake hub on an ephemeral port. Same + # gate and same RUN_SERIAL treatment as the suite above. + vllm_cpp_add_test(test_downloader vllm/transformers_utils/test_downloader.cpp) + vllm_cpp_add_test(test_model_resolver + vllm/transformers_utils/test_model_resolver.cpp) vllm_cpp_add_test(test_openai_api_server vllm/entrypoints/openai/test_api_server.cpp) # /v1/audio/transcriptions dispatch + socket smoke run against the REAL # library transcription seam on the committed parakeet_e2e fixture @@ -1545,6 +1551,7 @@ if(VLLM_CPP_SERVER) # these two against each other, not against the CPU-hog tests that actually # cause the starvation.) set_tests_properties(test_openai_api_server test_openai_conformance test_hf_hub + test_downloader test_model_resolver PROPERTIES RUN_SERIAL ON) # SERVE-RECIPE-ARGS (#606): the accepted-and-inert serve-argument seam. It # drives the REAL VllmServerMain (owned by the library behind the same @@ -1553,6 +1560,18 @@ if(VLLM_CPP_SERVER) vllm_cpp_add_test(test_serve_recipe_args vllm/entrypoints/openai/test_serve_recipe_args.cpp) target_include_directories(test_serve_recipe_args PRIVATE ${CMAKE_SOURCE_DIR}/src) + # ENG-HF-MODEL-DOWNLOAD W4 (#1280): THE REACHABILITY GATE. It enters at the + # REAL VllmServerMain with `--model org/repo` and an in-process fake hub, so + # it needs src/ on the include path for server_main.h, exactly as the suite + # above does, and the committed llama_embed_e2e checkpoint as the payload the + # hub serves. It binds a port and forks a server, so it is RUN_SERIAL for the + # same starvation reason as test_openai_api_server. + vllm_cpp_add_test(test_serve_hf_model + vllm/entrypoints/openai/test_serve_hf_model.cpp) + target_include_directories(test_serve_hf_model PRIVATE ${CMAKE_SOURCE_DIR}/src) + target_compile_definitions(test_serve_hf_model PRIVATE + VLLM_LLAMA_EMBED_FIXTURE_DIR="${CMAKE_SOURCE_DIR}/tests/vllm/models/fixtures/llama_embed_e2e") + set_tests_properties(test_serve_hf_model PROPERTIES RUN_SERIAL ON) # ENG-MM-INPUT-PIPELINE wave L2 (#607): --language-model-only / # --limit-mm-per-prompt. Same shape and the same reason as the row above — it # re-execs the REAL VllmServerMain to observe the flags reaching the config and diff --git a/tests/vllm/entrypoints/openai/test_serve_hf_model.cpp b/tests/vllm/entrypoints/openai/test_serve_hf_model.cpp new file mode 100644 index 000000000..284dffa25 --- /dev/null +++ b/tests/vllm/entrypoints/openai/test_serve_hf_model.cpp @@ -0,0 +1,471 @@ +// ENG-HF-MODEL-DOWNLOAD W4 (#1280): THE REACHABILITY GATE. +// +// `.agents/reachability.md` asks two questions, and this file answers the +// second one. `tests/vllm/transformers_utils/test_model_resolver.cpp` proves +// the resolver WORKS; it constructs the call itself, so it would stay green +// with the production call site deleted. This file proves the resolver is +// REACHED: it enters at `VllmServerMain(argc, argv)`, which is what the C ABI's +// `vllm_server_main` and the `vllm-server` binary both call, hands it +// `--model org/repo` with `HF_ENDPOINT` aimed at an in-process fake hub, and +// then asks the running server to complete a request. +// +// Deleting the `ResolveModelArgument` call in `server_main.cpp` turns this red: +// the loader then opens `tiny/llama` as a relative directory, finds nothing, +// and the server never binds. +// +// WHY A CHILD PROCESS. `VllmServerMain` blocks in `listen`, and `ParseArgs` +// reports a bad argument through `Usage()`, which calls `std::exit`. The +// pattern is the one in `test_serve_recipe_args.cpp` and +// `test_serve_residency_config.cpp`, with `fork` plus `execv` instead of +// `popen` so the parent holds the child's process id and can stop the server +// once it has answered. The FAKE HUB LIVES IN THE PARENT, which is why the +// child is a fresh `execv` rather than a forked copy: a forked child would +// inherit no listening thread and there would be nothing to fetch from. +#include + +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "support/process_id.h" +#include "vllm/entrypoints/openai/server_main.h" + +namespace fs = std::filesystem; +using nlohmann::json; + +namespace { + +constexpr const char* kCommit = "1111111111111111111111111111111111111111"; +// The DEFAULT branch's commit, and the hub serves NOTHING under it. The +// checkpoint lives on `kRevision` instead, which is what makes `--revision` +// load bearing: a run that does not carry the flag resolves `main`, gets this +// commit, lists an empty tree and never finds a config.json, so the server does +// not bind. Without that the flag would be reached but unpinned, because the +// fake hub answered `main` and `--revision main` identically. +constexpr const char* kDefaultCommit = + "2222222222222222222222222222222222222222"; +constexpr const char* kRevision = "serving"; +constexpr const char* kRepoId = "tiny/llama"; + +std::string ReadAll(const fs::path& path) { + std::ifstream in(path, std::ios::binary); + return std::string((std::istreambuf_iterator(in)), + std::istreambuf_iterator()); +} + +// The committed `llama_embed_e2e` checkpoint is a real, tiny, loadable +// safetensors file, and it is 154 KB rather than a fabricated tensor set. It +// carries the `LlamaModel` (pooling) tensor names, so every name gains the +// `model.` prefix the `LlamaForCausalLM` loader reads. The DATA SECTION is +// copied verbatim and the offsets are unchanged, because they are relative to +// the start of that section and the header length is not part of them. +std::string CausalLmSafetensors(const fs::path& source) { + const std::string bytes = ReadAll(source); + REQUIRE(bytes.size() > 8); + uint64_t header_len = 0; + for (int i = 7; i >= 0; --i) { + header_len = (header_len << 8) | + static_cast(bytes[static_cast(i)]); + } + REQUIRE(header_len + 8 <= bytes.size()); + const json header = json::parse(bytes.substr(8, header_len)); + json renamed = json::object(); + for (const auto& item : header.items()) { + if (item.key() == "__metadata__") continue; + renamed["model." + item.key()] = item.value(); + } + std::string text = renamed.dump(); + while (text.size() % 8 != 0) text.push_back(' '); + std::string out(8, '\0'); + for (int i = 0; i < 8; ++i) { + out[static_cast(i)] = + static_cast((text.size() >> (8 * i)) & 0xff); + } + out += text; + out += bytes.substr(8 + header_len); + return out; +} + +// The shape of the committed fixture, spelled as a generation config. The +// tensor sizes come from the checkpoint and the loader refuses a mismatch, so +// these numbers are pinned by the file rather than chosen here. +std::string CausalLmConfig() { + json config; + config["architectures"] = json::array({"LlamaForCausalLM"}); + config["model_type"] = "llama"; + config["hidden_size"] = 64; + config["num_hidden_layers"] = 2; + config["num_attention_heads"] = 4; + config["num_key_value_heads"] = 2; + config["head_dim"] = 16; + config["intermediate_size"] = 128; + config["rms_norm_eps"] = 1e-05; + config["rope_theta"] = 500000.0; + config["vocab_size"] = 32; + config["max_position_embeddings"] = 128; + config["torch_dtype"] = "bfloat16"; + // No `lm_head.weight` in the checkpoint, so the output head aliases the + // embedding table. The loader skips the tensor on this setting. + config["tie_word_embeddings"] = true; + config["attention_bias"] = false; + return config.dump(); +} + +// The fake hub: refs, a recursive tree, and the bytes. It records the paths it +// was asked for, so the decoy assertion is a statement about the server. +class FakeHub { + public: + FakeHub() { + server_.Get("/api/models/(.*)/refs", + [this](const httplib::Request& req, httplib::Response& res) { + Record(req); + res.set_content( + std::string( + R"({"branches":[{"name":"main","targetCommit":")") + + kDefaultCommit + R"("},{"name":")" + kRevision + + R"(","targetCommit":")" + kCommit + + R"("}],"tags":[]})", + "application/json"); + }); + server_.Get("/api/models/(.*)/tree/(.*)", + [this](const httplib::Request& req, httplib::Response& res) { + Record(req); + // Only `kCommit` holds the checkpoint. The default branch's + // commit lists nothing, so a run that lost `--revision` + // fetches nothing and the server never binds. + if (req.path.find(kCommit) == std::string::npos) { + res.set_content("[]", "application/json"); + return; + } + res.set_content(Tree(), "application/json"); + }); + server_.Get("/(.*)/resolve/(.*)", + [this](const httplib::Request& req, httplib::Response& res) { + Record(req); + const std::string marker = + std::string("/resolve/") + kCommit + "/"; + const size_t at = req.path.find(marker); + if (at == std::string::npos) { + res.status = 404; + return; + } + const auto it = files_.find(req.path.substr(at + marker.size())); + if (it == files_.end()) { + res.status = 404; + return; + } + res.set_header("Accept-Ranges", "bytes"); + res.set_header("ETag", "\"etag\""); + if (req.method == "HEAD") { + res.set_header("Content-Length", + std::to_string(it->second.size())); + res.status = 200; + return; + } + res.status = 200; + res.set_content(it->second, "application/octet-stream"); + }); + port_ = server_.bind_to_any_port("127.0.0.1"); + thread_ = std::thread([this] { server_.listen_after_bind(); }); + server_.wait_until_ready(); + } + ~FakeHub() { + server_.stop(); + if (thread_.joinable()) thread_.join(); + } + FakeHub(const FakeHub&) = delete; + FakeHub& operator=(const FakeHub&) = delete; + + std::string endpoint() const { + return "http://127.0.0.1:" + std::to_string(port_) + "/"; + } + void add(const std::string& path, std::string bytes) { + files_[path] = std::move(bytes); + } + std::vector paths() const { + const std::lock_guard lock(mu_); + return paths_; + } + bool asked_for(const std::string& needle) const { + for (const std::string& path : paths()) { + if (path.find(needle) != std::string::npos) return true; + } + return false; + } + + private: + std::string Tree() const { + json out = json::array(); + for (const auto& entry : files_) { + json item; + item["type"] = "file"; + item["path"] = entry.first; + item["size"] = entry.second.size(); + out.push_back(item); + } + return out.dump(); + } + void Record(const httplib::Request& req) { + const std::lock_guard lock(mu_); + paths_.push_back(req.path); + } + + httplib::Server server_; + std::thread thread_; + int port_ = 0; + mutable std::mutex mu_; + std::vector paths_; + std::map files_; +}; + +// A port nothing is listening on, AND THE PROBE SOCKET IS CLOSED before the +// number is returned. +// +// The first version of this function bound with an `httplib::Server` and called +// `probe.stop()`. That released nothing. `Server::stop()` +// (`third_party/httplib/httplib.h:11460`) is guarded by `if (is_running_)`, this +// function never calls `listen_after_bind()`, so `is_running_` was false and the +// socket was never shut down, and `Server::~Server()` is `= default` and does not +// close it either. httplib sets `SO_REUSEPORT` (`httplib.h:9455`), so the forked +// child bound the SAME port successfully and the kernel then load balanced +// inbound connections between the orphan socket, which nobody ever accepted on, +// and the real server. +// +// Measured on this box with `ss -ltnp` during an unmutated run at head +// `e25d3d344`: two LISTEN rows on one port, one owned by the parent and one by +// the child, the parent's carrying `Recv-Q 1`, which is the hung request sitting +// in a backlog nobody drains. Three runs took 90.4 s, 240.4 s and 180.4 s, and +// the 240.4 s one FAILED. Every stall was an exact multiple of a client read +// timeout rather than an intermediate value, which is what a contended box would +// have given. +// +// A plain POSIX socket is used rather than an httplib one because the close has +// to be unconditional. +int FreePort() { + const int fd = ::socket(AF_INET, SOCK_STREAM, 0); + REQUIRE(fd >= 0); + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_addr.s_addr = ::htonl(INADDR_LOOPBACK); + addr.sin_port = 0; + REQUIRE(::bind(fd, reinterpret_cast(&addr), sizeof(addr)) == 0); + socklen_t length = sizeof(addr); + REQUIRE(::getsockname(fd, reinterpret_cast(&addr), &length) == 0); + const int port = static_cast(::ntohs(addr.sin_port)); + REQUIRE(::close(fd) == 0); + return port; +} + +std::vector SplitOnSpaces(const std::string& text) { + std::vector out; + std::string current; + for (const char c : text) { + if (c == ' ') { + if (!current.empty()) out.push_back(current); + current.clear(); + } else { + current.push_back(c); + } + } + if (!current.empty()) out.push_back(current); + return out; +} + +} // namespace + +// THE CHILD. Skip-decorated, so a normal run never executes it; it runs only +// when the parent below re-execs this binary by name. +TEST_CASE("serve_hf_model_child" * doctest::skip()) { + const char* raw = std::getenv("VLLM_TEST_SERVE_ARGS"); + REQUIRE(raw != nullptr); + std::vector args{"vllm-server"}; + for (std::string& token : SplitOnSpaces(raw)) args.push_back(std::move(token)); + std::vector argv; + argv.reserve(args.size()); + for (std::string& arg : args) argv.push_back(arg.data()); + const int rc = vllm::entrypoints::openai::VllmServerMain( + static_cast(argv.size()), argv.data()); + std::cout << "SERVE_RC=" << rc << "\n" << std::flush; + std::exit(rc); +} + +TEST_CASE("serve: --model org/repo FETCHES the checkpoint and completes a request") { + const fs::path fixture(VLLM_LLAMA_EMBED_FIXTURE_DIR); + REQUIRE(fs::is_regular_file(fixture / "model.safetensors")); + + FakeHub hub; + const std::string weights = CausalLmSafetensors(fixture / "model.safetensors"); + hub.add("config.json", CausalLmConfig()); + hub.add("tokenizer.json", ReadAll(fixture / "tokenizer.json")); + hub.add("model.safetensors", weights); + hub.add("model.safetensors.index.json", + R"({"metadata":{"total_size":1},"weight_map":)" + R"({"model.embed_tokens.weight":"model.safetensors"}})"); + // The duplicate-format decoy a published checkpoint carries. Nothing must ask + // for it. + hub.add("original/model.safetensors", weights); + + const fs::path home = + fs::temp_directory_path() / + ("vllm_serve_hf_model_" + std::to_string(vllm_test::ProcessId())); + fs::remove_all(home); + fs::create_directories(home); + + // `--download-dir` IS the directory that holds the `models--org--repo` + // folders, which is how vLLM hands it to `snapshot_download(cache_dir=...)` + // (`config/model.py:183`). It is a sibling of `hub/`, never inside it, so the + // assertions at the end can tell the two apart. + const fs::path download_dir = home / "dl"; + fs::create_directories(download_dir); + + const int port = FreePort(); + // `--max-num-seqs 4` is a HARNESS setting, not a product one: the HTTP worker + // pool is sized from it plus four (`ApiServer::kControlWorkerHeadroom`), so + // the default 32 starts a 36-thread pool to serve one four-token request. + // It is NOT why this case used to be slow. That was a leaked listening socket + // in `FreePort`, and the note there records the measurement. + // + // `--revision` and `--download-dir` are here because they are PRODUCTION + // FLAGS with a production effect, and until they were on this line the only + // thing that reached `ParseArgs` for either of them was nothing at all: both + // `ParseArgs` branches could be deleted with every suite still green, because + // the unit cases set `ModelResolveOptions` by hand. They are load bearing + // now: the checkpoint is published on a non-default branch, and the cache + // lands under `--download-dir` rather than under `HF_HOME`. + const std::string serve_args = std::string("--model ") + kRepoId + + " --revision " + kRevision + + " --download-dir " + download_dir.string() + + " --port " + std::to_string(port) + + " --host 127.0.0.1 --max-model-len 64" + " --block-size 16 --max-num-seqs 4" + " --disable-log-requests"; + + char exe[4096]; + const ssize_t n = ::readlink("/proc/self/exe", exe, sizeof(exe) - 1); + REQUIRE(n > 0); + exe[n] = '\0'; + + ::setenv("VLLM_TEST_SERVE_ARGS", serve_args.c_str(), 1); + ::setenv("HF_ENDPOINT", hub.endpoint().c_str(), 1); + ::setenv("HF_HOME", home.c_str(), 1); + ::unsetenv("HF_HUB_OFFLINE"); + ::unsetenv("HF_TOKEN"); + + const pid_t pid = ::fork(); + REQUIRE(pid >= 0); + if (pid == 0) { + const char* child_argv[] = {exe, "--no-skip", + "--test-case=serve_hf_model_child", nullptr}; + ::execv(exe, const_cast(child_argv)); + std::_Exit(127); + } + + // Wait for the server to bind. A checkpoint this small loads in well under a + // second on any host, and the ceiling exists so a failure is a failure rather + // than a hung suite. + httplib::Client client("http://127.0.0.1:" + std::to_string(port)); + client.set_connection_timeout(1, 0); + client.set_read_timeout(30, 0); + bool healthy = false; + for (int attempt = 0; attempt < 600 && !healthy; ++attempt) { + int status = 0; + if (::waitpid(pid, &status, WNOHANG) == pid) break; // the child died + const httplib::Result health = client.Get("/health"); + healthy = health && health->status == 200; + if (!healthy) std::this_thread::sleep_for(std::chrono::milliseconds(100)); + } + + std::string completion_body; + std::string completion_error; + int completion_status = 0; + if (healthy) { + const json request = {{"model", kRepoId}, + {"prompt", "abc"}, + {"max_tokens", 4}, + {"temperature", 0}}; + // ONE ATTEMPT. There WAS a retry loop here, three attempts whenever the + // exchange did not complete, and it was absorbing the leaked probe socket + // that `FreePort` above now closes. With the leak gone the completion + // answers in 0.2 to 0.4 s and there is nothing left for a retry to absorb, + // so a second attempt could only hide the next defect the way this one hid + // that one. Sixty seconds of dead time read as a green run with no output. + // + // A FRESH client rather than the health poll's. That poll leaves a + // keep-alive socket the server may close on its own timeout, and a reused + // dead socket reports as a transport error that reads exactly like a + // refusal. + httplib::Client caller("http://127.0.0.1:" + std::to_string(port)); + caller.set_connection_timeout(5, 0); + caller.set_read_timeout(60, 0); + const httplib::Result answer = + caller.Post("/v1/completions", request.dump(), "application/json"); + if (answer) { + completion_status = answer->status; + completion_body = answer->body; + } else { + completion_error = httplib::to_string(answer.error()); + } + } + + ::kill(pid, SIGKILL); + int status = 0; + ::waitpid(pid, &status, 0); + + std::string joined; + for (const std::string& path : hub.paths()) joined += path + "\n"; + INFO("hub paths:\n" << joined); + INFO("completion: " << completion_body); + INFO("transport error: " << completion_error); + + // THE SERVER BOOTED FROM A CHECKPOINT IT FETCHED. Nothing was on disk when + // the process started: `HF_HOME` was an empty scratch directory. + REQUIRE(healthy); + REQUIRE(completion_status == 200); + const json completion = json::parse(completion_body); + REQUIRE(completion.contains("choices")); + REQUIRE(completion["choices"].size() == 1); + CHECK(completion["choices"][0]["finish_reason"] == "length"); + CHECK(completion["usage"]["completion_tokens"] == 4); + // The served name is what the user typed, not the commit directory the cache + // happens to hold. + CHECK(completion["model"] == kRepoId); + + // The fetch went through the hub, index driven, and never touched the decoy. + CHECK(hub.asked_for("/api/models/tiny/llama/refs")); + CHECK(hub.asked_for("/resolve/" + std::string(kCommit) + "/model.safetensors")); + CHECK_FALSE(hub.asked_for("original")); + + // And the cache holds the HuggingFace layout, so a second run is offline. + // + // IT LANDED UNDER `--download-dir`, NOT UNDER `HF_HOME`, which is the + // statement that `--download-dir` reached the resolver rather than being + // parsed and dropped. Deleting its `ParseArgs` branch puts the tree back + // under `HF_HOME/hub` and turns both of these red. + CHECK(fs::is_regular_file(download_dir / "models--tiny--llama" / "snapshots" / + kCommit / "config.json")); + CHECK_FALSE(fs::exists(home / "hub" / "models--tiny--llama")); + // And it is the `--revision` commit. `main` names `kDefaultCommit`, under + // which this hub lists nothing, so a run that lost the flag never gets this + // far: it fails at `REQUIRE(healthy)` above. + CHECK_FALSE(fs::exists(download_dir / "models--tiny--llama" / "snapshots" / + kDefaultCommit)); + + fs::remove_all(home); +} diff --git a/tests/vllm/transformers_utils/test_downloader.cpp b/tests/vllm/transformers_utils/test_downloader.cpp new file mode 100644 index 000000000..2a39f9392 --- /dev/null +++ b/tests/vllm/transformers_utils/test_downloader.cpp @@ -0,0 +1,484 @@ +// ENG-HF-MODEL-DOWNLOAD W3 (#1280): the byte transport. +// +// Every case runs against an IN-PROCESS FAKE HUB, a real `httplib::Server` on +// an ephemeral port reached over plain hypertext transfer protocol, following +// `tests/vllm/transformers_utils/test_hf_hub.cpp`. There is no TLS here on +// purpose: TLS has its own instruments in W5, and a hermetic test that speaks +// plain HTTP proves resume, truncation and integrity, not transport security. +// +// The hub RECORDS the request headers it saw. "The Range header was sent" and +// "the decoy was never requested" are both statements about what the server +// received, and a client-side mock cannot make either of them. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "support/process_id.h" +#include "vllm/transformers_utils/downloader.h" +#include "vllm/transformers_utils/hf_hub.h" + +namespace fs = std::filesystem; +using vllm::transformers_utils::HfDownloadOptions; +using vllm::transformers_utils::HfDownloadResult; +using vllm::transformers_utils::HfFileShape; +using vllm::transformers_utils::HfHubOptions; +using vllm::transformers_utils::HfInstallDownloadInterruptHandler; +using vllm::transformers_utils::HfRemoteFile; +using vllm::transformers_utils::HfRepoLock; +using vllm::transformers_utils::HfResetDownloadInterrupt; +using vllm::transformers_utils::HfShapeForPath; +using vllm::transformers_utils::HfVerifyFileShape; +using vllm::transformers_utils::HubDownloadFile; +using vllm::transformers_utils::HubProbeFile; + +namespace { + +class TempDir { + public: + TempDir() { + static std::atomic counter{0}; + path_ = fs::temp_directory_path() / + ("vllm_downloader_test_" + std::to_string(vllm_test::ProcessId()) + + "_" + std::to_string(counter.fetch_add(1))); + fs::remove_all(path_); + fs::create_directories(path_); + } + ~TempDir() { + std::error_code ec; + fs::remove_all(path_, ec); + } + TempDir(const TempDir&) = delete; + TempDir& operator=(const TempDir&) = delete; + const fs::path& path() const { return path_; } + + private: + fs::path path_; +}; + +// A valid single-tensor safetensors blob: 8 length bytes, the header, then +// exactly `data_end` payload bytes. `extra_payload` appends bytes the header +// does not account for, which is the shape the data-end rule refuses. +std::string SafetensorsBlob(size_t payload_bytes, size_t extra_payload = 0) { + nlohmann::json header; + header["t"] = {{"dtype", "U8"}, + {"shape", {static_cast(payload_bytes)}}, + {"data_offsets", {0, static_cast(payload_bytes)}}}; + std::string text = header.dump(); + while (text.size() % 8 != 0) text.push_back(' '); + std::string out(8, '\0'); + for (int i = 0; i < 8; ++i) { + out[static_cast(i)] = + static_cast((text.size() >> (8 * i)) & 0xff); + } + out += text; + for (size_t i = 0; i < payload_bytes; ++i) { + out.push_back(static_cast('A' + static_cast(i % 26))); + } + out.append(extra_payload, 'Z'); + return out; +} + +// The fake hub, serving ONE body at ONE address, with the three answers a +// transport has to survive. +class ByteHub { + public: + ByteHub() { + // httplib routes HEAD to the GET handler + // (third_party/httplib/httplib.h:12183) and writes no body for it, so both + // verbs are answered here and the method is read from the request. + server_.Get("/file", [this](const httplib::Request& req, + httplib::Response& res) { + Record(req); + res.set_header("ETag", "\"" + etag_ + "\""); + res.set_header("Accept-Ranges", accepts_ranges_ ? "bytes" : "none"); + if (req.method == "HEAD") { + res.set_header("Content-Length", std::to_string(body_.size())); + res.status = 200; + return; + } + + if (truncate_after_ > 0) { + // A body SHORTER than the length its own answer declares. httplib + // reports some of these as a read error and others as a short success, + // so this is the shape the byte count in the downloader exists for. + const std::string prefix = body_.substr(0, truncate_after_); + const size_t declared = body_.size(); + res.set_content_provider( + declared, "application/octet-stream", + [prefix](size_t offset, size_t, httplib::DataSink& sink) { + if (offset == 0) { + sink.write(prefix.data(), prefix.size()); + return true; + } + return false; // abort: the connection closes early. + }); + return; + } + + const bool ranged = req.has_header("Range") && !ignore_range_; + // THE FULL BODY IS SET IN BOTH ARMS. httplib slices it itself when the + // request carried a range AND the status is 206 + // (`Server::apply_ranges`, third_party/httplib/httplib.h:12231), and it + // writes the `Content-Range` header from that slice. A handler that + // pre-sliced would be sliced a SECOND time, which is a fixture defect + // that reads exactly like a resume bug in the client: this suite + // measured a 12 byte tail where 24 were due. + res.status = ranged ? 206 : 200; + res.set_content(body_, "application/octet-stream"); + }); + port_ = server_.bind_to_any_port("127.0.0.1"); + thread_ = std::thread([this] { server_.listen_after_bind(); }); + server_.wait_until_ready(); + } + ~ByteHub() { + server_.stop(); + if (thread_.joinable()) thread_.join(); + } + ByteHub(const ByteHub&) = delete; + ByteHub& operator=(const ByteHub&) = delete; + + std::string url() const { + return "http://127.0.0.1:" + std::to_string(port_) + "/file"; + } + void set_body(std::string body) { body_ = std::move(body); } + void set_ignore_range(bool value) { ignore_range_ = value; } + void set_accepts_ranges(bool value) { accepts_ranges_ = value; } + void set_truncate_after(size_t value) { truncate_after_ = value; } + const std::string& body() const { return body_; } + + int request_count() const { return requests_.load(); } + std::vector range_headers() const { + const std::lock_guard lock(mu_); + return ranges_; + } + + private: + void Record(const httplib::Request& req) { + requests_.fetch_add(1); + const std::lock_guard lock(mu_); + if (req.has_header("Range")) ranges_.push_back(req.get_header_value("Range")); + } + + httplib::Server server_; + std::thread thread_; + int port_ = 0; + std::atomic requests_{0}; + mutable std::mutex mu_; + std::vector ranges_; + std::string body_ = "0123456789abcdefghijklmnopqrstuvwxyz"; + std::string etag_ = "abc123"; + bool ignore_range_ = false; + bool accepts_ranges_ = true; + size_t truncate_after_ = 0; +}; + +HfHubOptions OptionsFor() { + HfHubOptions opts; + opts.connect_timeout_seconds = 5; + opts.read_timeout_seconds = 5; + return opts; +} + +HfDownloadOptions DownloadOptionsFor() { + HfDownloadOptions opts; + opts.hub = OptionsFor(); + return opts; +} + +std::string ReadAll(const fs::path& path) { + std::ifstream in(path, std::ios::binary); + return std::string((std::istreambuf_iterator(in)), + std::istreambuf_iterator()); +} + +void WriteAll(const fs::path& path, const std::string& bytes) { + std::ofstream out(path, std::ios::binary | std::ios::trunc); + out.write(bytes.data(), static_cast(bytes.size())); +} + +std::string MessageOf(const std::function& body) { + try { + body(); + } catch (const std::exception& e) { + return e.what(); + } + return std::string(); +} + +} // namespace + +TEST_CASE("downloader: a HEAD probe reads the size, the entity tag and range support") { + ByteHub hub; + const HfRemoteFile info = HubProbeFile(hub.url(), OptionsFor()); + REQUIRE(info.size.has_value()); + CHECK(*info.size == hub.body().size()); + // The quotes are transport spelling and must not reach a blob file name. + CHECK(info.etag == "abc123"); + CHECK(info.accepts_ranges); + + hub.set_accepts_ranges(false); + CHECK_FALSE(HubProbeFile(hub.url(), OptionsFor()).accepts_ranges); +} + +TEST_CASE("downloader: a complete transfer lands the bytes and leaves no .incomplete") { + ByteHub hub; + TempDir dir; + const fs::path dest = dir.path() / "blob"; + + const HfDownloadResult result = + HubDownloadFile(hub.url(), dest, hub.body().size(), HfFileShape::kOpaque, + DownloadOptionsFor()); + CHECK(result.bytes_written == hub.body().size()); + CHECK_FALSE(result.already_present); + CHECK(ReadAll(dest) == hub.body()); + // The `.incomplete` name is what a partial transfer occupies. Its absence is + // the proof the rename happened rather than a copy. + CHECK_FALSE(fs::exists(fs::path(dest.string() + ".incomplete"))); + + // A SECOND call opens no socket at all. That is what makes a warm cache free. + const int before = hub.request_count(); + const HfDownloadResult again = + HubDownloadFile(hub.url(), dest, hub.body().size(), HfFileShape::kOpaque, + DownloadOptionsFor()); + CHECK(again.already_present); + CHECK(again.bytes_written == 0); + CHECK(hub.request_count() == before); +} + +TEST_CASE("downloader: an interrupted transfer RESUMES with a Range header") { + ByteHub hub; + TempDir dir; + const fs::path dest = dir.path() / "blob"; + const fs::path partial = fs::path(dest.string() + ".incomplete"); + + // Stand in for a transfer that stopped a third of the way through. + const size_t already = 12; + WriteAll(partial, hub.body().substr(0, already)); + + const HfDownloadResult result = + HubDownloadFile(hub.url(), dest, hub.body().size(), HfFileShape::kOpaque, + DownloadOptionsFor()); + CHECK(result.resumed); + // The TAIL was transferred, not the whole file. A test that only checked the + // final bytes would pass on a client that threw the partial file away. + CHECK(result.bytes_written == hub.body().size() - already); + + const std::vector ranges = hub.range_headers(); + REQUIRE_FALSE(ranges.empty()); + CHECK(ranges.back() == "bytes=12-"); + // And the FINAL bytes are the whole body, not the tail written twice. + CHECK(ReadAll(dest) == hub.body()); +} + +TEST_CASE("downloader: a 200 answer to a range request is REFUSED, never appended") { + // THE DELIBERATE DIVERGENCE FROM llama.cpp `common/download.cpp:222-235 @ + // b10451`, which warns and continues. Appending a whole body onto a partial + // file writes the first N bytes twice, and a token gate cannot see a corrupt + // weight: the model still emits tokens. + ByteHub hub; + hub.set_ignore_range(true); + TempDir dir; + const fs::path dest = dir.path() / "blob"; + const fs::path partial = fs::path(dest.string() + ".incomplete"); + WriteAll(partial, hub.body().substr(0, 12)); + + const std::string message = MessageOf([&] { + HubDownloadFile(hub.url(), dest, hub.body().size(), HfFileShape::kOpaque, + DownloadOptionsFor()); + }); + INFO("refusal: " << message); + REQUIRE_FALSE(message.empty()); + CHECK(message.find("HTTP 200") != std::string::npos); + CHECK(message.find("206") != std::string::npos); + // The destination name was never taken, so a later run cannot read the + // wreckage as a cache hit. + CHECK_FALSE(fs::exists(dest)); + // And the partial file is still only the 12 bytes it started as: nothing was + // appended before the refusal. + CHECK(fs::file_size(partial) == 12); +} + +TEST_CASE("downloader: a body shorter than its Content-Length is refused") { + ByteHub hub; + hub.set_truncate_after(10); + TempDir dir; + const fs::path dest = dir.path() / "blob"; + + const std::string message = MessageOf([&] { + HubDownloadFile(hub.url(), dest, hub.body().size(), HfFileShape::kOpaque, + DownloadOptionsFor()); + }); + INFO("refusal: " << message); + REQUIRE_FALSE(message.empty()); + // The refusal must come from the BYTE COUNT, not from whatever the transport + // happened to report. Measured: with the count checked after httplib's own + // verdict, deleting the count left this case green, because the generic + // transport message satisfied "it refused". + CHECK(message.find("truncated") != std::string::npos); + CHECK(message.find("delivered 10") != std::string::npos); + CHECK_FALSE(fs::exists(dest)); +} + +TEST_CASE("downloader: a safetensors whose data end misses the file size is refused") { + // `8 + header_len + max(data_offsets[1]) == file_size`. The body below is a + // well-formed header over 64 payload bytes with 16 bytes nobody accounts for + // appended, which is what a resumed-onto-a-200 transfer leaves behind and + // what a length field cannot see. + ByteHub hub; + hub.set_body(SafetensorsBlob(64, /*extra_payload=*/16)); + TempDir dir; + const fs::path dest = dir.path() / "model.safetensors"; + + CHECK(HfShapeForPath("model.safetensors") == HfFileShape::kSafetensors); + const std::string message = MessageOf([&] { + HubDownloadFile(hub.url(), dest, hub.body().size(), + HfFileShape::kSafetensors, DownloadOptionsFor()); + }); + INFO("refusal: " << message); + REQUIRE_FALSE(message.empty()); + CHECK(message.find("safetensors") != std::string::npos); + // The transfer matched every length the transport reported. Only the + // structural rule can refuse it, which is why the row does not accept an + // opaque remote field as proof. + CHECK_FALSE(fs::exists(dest)); + + // The same blob WITHOUT the unaccounted tail is accepted. + ByteHub good; + good.set_body(SafetensorsBlob(64)); + const fs::path ok = dir.path() / "good.safetensors"; + HubDownloadFile(good.url(), ok, good.body().size(), HfFileShape::kSafetensors, + DownloadOptionsFor()); + CHECK(fs::exists(ok)); +} + +TEST_CASE("downloader: the shape a path declares by its extension") { + CHECK(HfShapeForPath("model.safetensors") == HfFileShape::kSafetensors); + CHECK(HfShapeForPath("dir/Model-Q4_K_M.gguf") == HfFileShape::kGguf); + CHECK(HfShapeForPath("config.json") == HfFileShape::kOpaque); + CHECK(HfShapeForPath("tokenizer.model") == HfFileShape::kOpaque); +} + +TEST_CASE("downloader: what an https address does, on this build's TLS state") { + // BOTH ARMS ASSERT. The first version of this case wrapped its whole body in + // `#ifndef CPPHTTPLIB_OPENSSL_SUPPORT`, so the moment W5 defines that macro + // the case would have compiled to nothing and reported `assertions: 0`, which + // this tree records as a skip wearing a pass. The preprocessor now selects + // WHICH statement is made, never whether one is made. + // + // The address is LOOPBACK on a port nothing listens on, not + // `huggingface.co`. Under a no-TLS build the scheme is refused before any + // socket opens, so the host never mattered; under a TLS build it would be a + // real network call, and a hermetic suite must not make one. + TempDir dir; + const std::string message = MessageOf([&] { + HubProbeFile("https://127.0.0.1:1/org/repo/resolve/main/config.json", + OptionsFor()); + }); + INFO("refusal: " << message); + REQUIRE_FALSE(message.empty()); +#ifdef CPPHTTPLIB_OPENSSL_SUPPORT + // This build SPEAKS https, so the refusal is the connection failing and must + // NOT name the build options: a message telling a working build to rebuild + // itself sends the reader after the wrong thing. + CHECK(message.find("cannot speak HTTPS") == std::string::npos); + CHECK(message.find("VLLM_CPP_OPENSSL") == std::string::npos); + CHECK(message.find("VLLM_CPP_BUILD_BORINGSSL") == std::string::npos); +#else + CHECK(message.find("VLLM_CPP_HF_DOWNLOAD") != std::string::npos); + CHECK(message.find("VLLM_CPP_OPENSSL") != std::string::npos); + CHECK(message.find("VLLM_CPP_BUILD_BORINGSSL") != std::string::npos); +#endif +} + +TEST_CASE("downloader: HF_HUB_OFFLINE opens no socket") { + ByteHub hub; + TempDir dir; + HfDownloadOptions opts = DownloadOptionsFor(); + opts.hub.offline = true; + + const int before = hub.request_count(); + const std::string message = MessageOf([&] { + HubDownloadFile(hub.url(), dir.path() / "blob", std::nullopt, + HfFileShape::kOpaque, opts); + }); + INFO("refusal: " << message); + CHECK(message.find("HF_HUB_OFFLINE") != std::string::npos); + CHECK(hub.request_count() == before); +} + +TEST_CASE("downloader: SIGINT cancels the transfer and KEEPS the partial file") { + ByteHub hub; + TempDir dir; + const fs::path dest = dir.path() / "blob"; + // The runner installs its OWN SIGINT handler and reports a chained signal as + // a crashed case, and the production handler chains on purpose so that a + // second Ctrl-C still ends the process. Parking SIGINT on SIG_IGN for the + // duration makes the chain a no-op, so the REAL handler is the thing under + // test and the runner is not signalled. The previous handler is put back + // before the case returns. + void (*runner_handler)(int) = std::signal(SIGINT, SIG_IGN); + HfInstallDownloadInterruptHandler(); + std::raise(SIGINT); + CHECK(vllm::transformers_utils::HfDownloadInterrupted()); + + const std::string message = MessageOf([&] { + HubDownloadFile(hub.url(), dest, hub.body().size(), HfFileShape::kOpaque, + DownloadOptionsFor()); + }); + HfResetDownloadInterrupt(); + std::signal(SIGINT, runner_handler); + INFO("refusal: " << message); + REQUIRE_FALSE(message.empty()); + CHECK(message.find("SIGINT") != std::string::npos); + CHECK_FALSE(fs::exists(dest)); + // The partial file survives, so the next run resumes rather than starting + // again. That is the whole point of cancelling rather than deleting. + CHECK(fs::exists(fs::path(dest.string() + ".incomplete"))); +} + +TEST_CASE("downloader: the per-repository lock is taken beside the repository") { + TempDir dir; + const fs::path repo = dir.path() / "models--org--repo"; + { + const HfRepoLock lock(repo); + CHECK(lock.held()); + CHECK(lock.path() == fs::path(repo.string() + ".lock")); + // It is NOT inside the repository directory, so the cache walk never has to + // filter it out of a snapshot listing. + CHECK(lock.path().parent_path() == dir.path()); + } + // An empty repository path is a host with no cache. It takes no lock rather + // than locking a file named by the empty string. + const HfRepoLock none{fs::path()}; + CHECK_FALSE(none.held()); +} + +TEST_CASE("downloader: HfVerifyFileShape refuses a truncated safetensors on disk") { + TempDir dir; + const fs::path file = dir.path() / "x.safetensors"; + const std::string blob = SafetensorsBlob(64); + WriteAll(file, blob.substr(0, blob.size() - 8)); + const std::string message = + MessageOf([&] { HfVerifyFileShape(file, HfFileShape::kSafetensors); }); + INFO("refusal: " << message); + REQUIRE_FALSE(message.empty()); + + WriteAll(file, blob); + HfVerifyFileShape(file, HfFileShape::kSafetensors); // no throw + // An opaque file is never opened, whatever it holds. + WriteAll(file, "not a safetensors at all"); + HfVerifyFileShape(file, HfFileShape::kOpaque); +} diff --git a/tests/vllm/transformers_utils/test_model_resolver.cpp b/tests/vllm/transformers_utils/test_model_resolver.cpp new file mode 100644 index 000000000..a9e683cf8 --- /dev/null +++ b/tests/vllm/transformers_utils/test_model_resolver.cpp @@ -0,0 +1,590 @@ +// ENG-HF-MODEL-DOWNLOAD W4 (#1280): the `--model` grammar. +// +// Every case runs against an IN-PROCESS FAKE HUB, a real `httplib::Server` on +// an ephemeral port reached over plain hypertext transfer protocol through +// `HF_ENDPOINT`, following `tests/vllm/transformers_utils/test_hf_hub.cpp`. +// +// THE HUB RECORDS EVERY PATH IT WAS ASKED FOR, and two of the cases below are +// statements about a request that was NOT made: a local path resolves with the +// hub receiving nothing, and a snapshot fetch never asks for the decoy +// `original/model.safetensors`. A downloaded-file COUNT bounds neither: it +// cannot say which file was skipped, and it reads the same whether the decoy +// was skipped or a real shard was. +// +// Reachability is NOT proven here. This file calls the resolver directly, so it +// cannot see a deleted call site in `server_main.cpp`. That is +// `tests/vllm/entrypoints/openai/test_serve_hf_model.cpp`, which enters through +// `vllm_server_main`. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include "support/process_id.h" +#include "support/test_env.h" +#include "vllm/gguf_builder.h" +#include "vllm/transformers_utils/model_resolver.h" + +namespace fs = std::filesystem; +using vllm::transformers_utils::IsRepoRootFile; +using vllm::transformers_utils::ModelReference; +using vllm::transformers_utils::ModelResolveOptions; +using vllm::transformers_utils::ParsedModelReference; +using vllm::transformers_utils::ParseModelReference; +using vllm::transformers_utils::ResolveModelPath; +using vllm::transformers_utils::SelectWeightFiles; + +namespace { + +constexpr const char* kCommit = "1111111111111111111111111111111111111111"; + +class TempDir { + public: + TempDir() { + static std::atomic counter{0}; + path_ = fs::temp_directory_path() / + ("vllm_model_resolver_test_" + + std::to_string(vllm_test::ProcessId()) + "_" + + std::to_string(counter.fetch_add(1))); + fs::remove_all(path_); + fs::create_directories(path_); + } + ~TempDir() { + std::error_code ec; + fs::remove_all(path_, ec); + } + TempDir(const TempDir&) = delete; + TempDir& operator=(const TempDir&) = delete; + const fs::path& path() const { return path_; } + + private: + fs::path path_; +}; + +// A minimal but VALID GGUF: the structural proof runs the tree's own reader, so +// a fixture of random bytes would be refused for a reason no case is about. +std::string TinyGguf() { + gguf_test::GgufModelBuilder builder; + builder.AddKv(gguf_test::StrKv("general.architecture", "llama")); + builder.AddTensor("token_embd.weight", {4, 2}, /*F32=*/0, + std::string(4 * 2 * 4, '\x01')); + return builder.Build(); +} + +// A VALID single-tensor safetensors blob. The structural proof runs on every +// `.safetensors` the resolver fetches, so a fixture of arbitrary bytes would be +// refused for a reason none of these cases is about. +std::string TinySafetensors(const std::string& marker) { + nlohmann::json header; + header["t"] = {{"dtype", "U8"}, + {"shape", {marker.size()}}, + {"data_offsets", {0, marker.size()}}}; + std::string text = header.dump(); + while (text.size() % 8 != 0) text.push_back(' '); + std::string out(8, '\0'); + for (int i = 0; i < 8; ++i) { + out[static_cast(i)] = + static_cast((text.size() >> (8 * i)) & 0xff); + } + return out + text + marker; +} + +// The repository the fake hub serves. `files` maps a repository-relative path +// to its bytes. +class FakeHub { + public: + FakeHub() { + server_.Get("/api/models/(.*)/refs", + [this](const httplib::Request& req, httplib::Response& res) { + Record(req); + res.set_content( + std::string( + R"({"branches":[{"name":"main","targetCommit":")") + + kCommit + R"("}],"tags":[]})", + "application/json"); + }); + server_.Get("/api/models/(.*)/tree/(.*)", + [this](const httplib::Request& req, httplib::Response& res) { + Record(req); + res.set_content(Tree(), "application/json"); + }); + server_.Get("/(.*)/resolve/(.*)", + [this](const httplib::Request& req, httplib::Response& res) { + Record(req); + const std::string marker = + std::string("/resolve/") + kCommit + "/"; + const size_t at = req.path.find(marker); + if (at == std::string::npos) { + res.status = 404; + return; + } + const std::string rel = req.path.substr(at + marker.size()); + const auto it = files_.find(rel); + if (it == files_.end()) { + res.status = 404; + return; + } + res.set_header("Accept-Ranges", "bytes"); + res.set_header("ETag", "\"etag\""); + if (req.method == "HEAD") { + res.set_header("Content-Length", + std::to_string(it->second.size())); + res.status = 200; + return; + } + res.status = 200; + res.set_content(it->second, "application/octet-stream"); + }); + port_ = server_.bind_to_any_port("127.0.0.1"); + thread_ = std::thread([this] { server_.listen_after_bind(); }); + server_.wait_until_ready(); + } + ~FakeHub() { + server_.stop(); + if (thread_.joinable()) thread_.join(); + } + FakeHub(const FakeHub&) = delete; + FakeHub& operator=(const FakeHub&) = delete; + + std::string endpoint() const { + return "http://127.0.0.1:" + std::to_string(port_) + "/"; + } + void add(const std::string& path, std::string bytes) { + files_[path] = std::move(bytes); + } + // Give one listed file an object identifier. The hub reports one for a + // large-file entry and reports none for a small one, and which of the two a + // file gets decides the name its cache blob takes. + void add_oid(const std::string& path, std::string oid) { + oids_[path] = std::move(oid); + } + int request_count() const { return requests_.load(); } + std::vector paths() const { + const std::lock_guard lock(mu_); + return paths_; + } + bool asked_for(const std::string& needle) const { + for (const std::string& path : paths()) { + if (path.find(needle) != std::string::npos) return true; + } + return false; + } + + private: + std::string Tree() const { + nlohmann::json out = nlohmann::json::array(); + for (const auto& entry : files_) { + nlohmann::json item; + item["type"] = "file"; + item["path"] = entry.first; + item["size"] = entry.second.size(); + const auto oid = oids_.find(entry.first); + if (oid != oids_.end()) item["oid"] = oid->second; + out.push_back(item); + } + return out.dump(); + } + void Record(const httplib::Request& req) { + requests_.fetch_add(1); + const std::lock_guard lock(mu_); + paths_.push_back(req.path); + } + + httplib::Server server_; + std::thread thread_; + int port_ = 0; + std::atomic requests_{0}; + mutable std::mutex mu_; + std::vector paths_; + std::map files_; + std::map oids_; +}; + +// A scratch `HF_HOME` and `HF_ENDPOINT`, restored on the way out so one case +// cannot leak its hub into the next. +class HubEnvironment { + public: + HubEnvironment(const FakeHub& hub, const fs::path& home) { + const char* endpoint = std::getenv("HF_ENDPOINT"); + const char* hf_home = std::getenv("HF_HOME"); + const char* offline = std::getenv("HF_HUB_OFFLINE"); + saved_endpoint_ = endpoint == nullptr ? std::string() : endpoint; + saved_home_ = hf_home == nullptr ? std::string() : hf_home; + saved_offline_ = offline == nullptr ? std::string() : offline; + vllm_test::SetEnv("HF_ENDPOINT", hub.endpoint().c_str()); + vllm_test::SetEnv("HF_HOME", home.string().c_str()); + vllm_test::SetEnv("HF_HUB_OFFLINE", ""); + vllm_test::SetEnv("HF_TOKEN", ""); + } + ~HubEnvironment() { + vllm_test::SetEnv("HF_ENDPOINT", saved_endpoint_.c_str()); + vllm_test::SetEnv("HF_HOME", saved_home_.c_str()); + vllm_test::SetEnv("HF_HUB_OFFLINE", saved_offline_.c_str()); + } + HubEnvironment(const HubEnvironment&) = delete; + HubEnvironment& operator=(const HubEnvironment&) = delete; + + private: + std::string saved_endpoint_; + std::string saved_home_; + std::string saved_offline_; +}; + +std::string ReadAll(const fs::path& path) { + std::ifstream in(path, std::ios::binary); + return std::string((std::istreambuf_iterator(in)), + std::istreambuf_iterator()); +} + +std::string MessageOf(const std::function& body) { + try { + body(); + } catch (const std::exception& e) { + return e.what(); + } + return std::string(); +} + +// A repository shaped like a published safetensors checkpoint, DECOY included. +void PopulateSafetensorsRepo(FakeHub& hub) { + hub.add("config.json", R"({"architectures":["LlamaForCausalLM"]})"); + hub.add("tokenizer.json", R"({"version":"1.0"})"); + hub.add("tokenizer_config.json", R"({"model_max_length":128})"); + hub.add("README.md", "# a model card nobody needs to fetch"); + hub.add("model.safetensors", TinySafetensors("SHARD-ONE-BYTES")); + hub.add("model.safetensors.index.json", + R"({"metadata":{"total_size":15},)" + R"("weight_map":{"a":"model.safetensors","b":"model.safetensors"}})"); + // TWO DECOYS, because two independent guards keep them out and one decoy + // would leave whichever guard it does not reach ungated. + // + // `original/` is a duplicate-format copy a published checkpoint carries. It + // is kept out by the ROOT-ONLY rule, which is why the directory also holds a + // `config.json`: phase one fetches every configuration file it sees, so a + // rule that stopped filtering by depth would ask for this one. + hub.add("original/model.safetensors", + TinySafetensors("A-SECOND-COPY-NOBODY-WANTS")); + hub.add("original/config.json", R"({"architectures":["LlamaForCausalLM"]})"); + // `consolidated.safetensors` sits at the ROOT beside the sharded weights, the + // way Mistral publishes one, and only INDEX-DRIVEN selection keeps it out: it + // matches `*.safetensors` and the `weight_map` does not name it. + hub.add("consolidated.safetensors", + TinySafetensors("THE-UNSHARDED-COPY-NOBODY-WANTS")); +} + +} // namespace + +TEST_CASE("model resolver: the grammar, decided without a socket") { + TempDir dir; + fs::create_directories(dir.path() / "local-model"); + { std::ofstream out(dir.path() / "local.gguf", std::ios::binary); out << "GGUF"; } + + CHECK(ParseModelReference((dir.path() / "local-model").string()).kind == + ModelReference::kLocalDirectory); + CHECK(ParseModelReference((dir.path() / "local.gguf").string()).kind == + ModelReference::kLocalGgufFile); + + const ParsedModelReference snapshot = ParseModelReference("org/repo"); + CHECK(snapshot.kind == ModelReference::kHubSnapshot); + CHECK(snapshot.repo_id == "org/repo"); + + const ParsedModelReference gguf = ParseModelReference("org/repo:Q4_K_M"); + CHECK(gguf.kind == ModelReference::kHubGgufFile); + CHECK(gguf.repo_id == "org/repo"); + CHECK(gguf.tag == "Q4_K_M"); + + // THE LAST COLON, which is why a Windows path is not read as a tag. Splitting + // on the first colon would make `C` the repository and `\models\qwen` the + // quantization. + CHECK(ParseModelReference("C:\\models\\qwen").kind == + ModelReference::kUnrecognized); + // `org/repo:x` is a valid repository plus a tag; `org/repo:x:Q8_0` splits at + // the LAST colon, and `org/repo:x` is not a repository identifier, so the + // whole value falls through rather than silently naming a different tag. + CHECK(ParseModelReference("org/repo:x:Q8_0").kind == + ModelReference::kUnrecognized); + + CHECK(ParseModelReference("/no/such/path").kind == + ModelReference::kUnrecognized); + CHECK(ParseModelReference("").kind == ModelReference::kUnrecognized); + // Two slashes is not `org/repo`. + CHECK(ParseModelReference("a/b/c").kind == ModelReference::kUnrecognized); +} + +TEST_CASE("model resolver: a LOCAL path resolves and the hub receives NOTHING") { + FakeHub hub; + TempDir dir; + const HubEnvironment env(hub, dir.path() / "home"); + PopulateSafetensorsRepo(hub); + + const fs::path local = dir.path() / "local-model"; + fs::create_directories(local); + { std::ofstream out(local / "config.json"); out << "{}"; } + + ModelResolveOptions opts; + CHECK(ResolveModelPath(local.string(), opts) == local.string()); + + const fs::path gguf = dir.path() / "local.gguf"; + { std::ofstream out(gguf, std::ios::binary); out << TinyGguf(); } + CHECK(ResolveModelPath(gguf.string(), opts) == gguf.string()); + + // An unrecognized value comes back unchanged so the loader's existing error + // still fires on what the user typed. + CHECK(ResolveModelPath("/no/such/model", opts) == "/no/such/model"); + + // ZERO requests. A local path can never be shadowed by a network call. + CHECK(hub.request_count() == 0); +} + +TEST_CASE("model resolver: a snapshot fetch NEVER asks for the decoy under original/") { + FakeHub hub; + TempDir dir; + const HubEnvironment env(hub, dir.path() / "home"); + PopulateSafetensorsRepo(hub); + + ModelResolveOptions opts; + const std::string snapshot = ResolveModelPath("org/repo", opts); + + std::string joined; + for (const std::string& path : hub.paths()) joined += path + "\n"; + INFO("paths:\n" << joined); + + // THE ASSERTIONS THIS CASE EXISTS FOR, one per guard. A count of downloaded + // files would read the same whether the skipped file was a decoy or a real + // shard, and it could not say which guard did the skipping. + // + // The root-only rule keeps the duplicate-format directory out. + CHECK_FALSE(hub.asked_for("original")); + // Index-driven selection keeps the root-level second copy out. Removing it + // and globbing `*.safetensors` fetches this file too. + CHECK_FALSE(hub.asked_for("consolidated")); + + // And the snapshot is complete and usable. + CHECK(fs::is_regular_file(fs::path(snapshot) / "config.json")); + CHECK(fs::is_regular_file(fs::path(snapshot) / "tokenizer.json")); + CHECK(fs::is_regular_file(fs::path(snapshot) / "model.safetensors")); + CHECK(ReadAll(fs::path(snapshot) / "model.safetensors") == + TinySafetensors("SHARD-ONE-BYTES")); + CHECK(fs::path(snapshot).filename().string() == kCommit); + + // A SECOND run over the warm cache asks for no bytes at all: only the tree + // listing, because the reference is already recorded. + const size_t before = hub.paths().size(); + const std::string again = ResolveModelPath("org/repo", opts); + CHECK(again == snapshot); + int byte_requests = 0; + const std::vector after = hub.paths(); + for (size_t i = before; i < after.size(); ++i) { + if (after[i].find("/resolve/") != std::string::npos) byte_requests += 1; + } + CHECK(byte_requests == 0); +} + +TEST_CASE("model resolver: a repository with no config.json fails BEFORE the weights") { + FakeHub hub; + TempDir dir; + const HubEnvironment env(hub, dir.path() / "home"); + hub.add("README.md", "# a dataset, not a model"); + hub.add("model.safetensors", TinySafetensors(std::string(4096, 'W'))); + + ModelResolveOptions opts; + const std::string message = + MessageOf([&] { ResolveModelPath("org/repo", opts); }); + INFO("refusal: " << message); + REQUIRE_FALSE(message.empty()); + CHECK(message.find("config.json") != std::string::npos); + // THE POINT OF THE TWO PHASES: the weights were never asked for. A one-phase + // fetch would have paid for them before finding out. + CHECK_FALSE(hub.asked_for("model.safetensors")); +} + +TEST_CASE("model resolver: org/repo:Q4_K_M returns ONE gguf file") { + FakeHub hub; + TempDir dir; + const HubEnvironment env(hub, dir.path() / "home"); + const std::string gguf = TinyGguf(); + hub.add("README.md", "# quantizations"); + hub.add("Model-Q4_K_M.gguf", gguf); + hub.add("Model-Q8_0.gguf", gguf); + + ModelResolveOptions opts; + const std::string path = ResolveModelPath("org/repo:Q4_K_M", opts); + CHECK(fs::path(path).filename().string() == "Model-Q4_K_M.gguf"); + CHECK(fs::is_regular_file(path)); + CHECK(ReadAll(path) == gguf); + // The other quantization was never fetched. `org/repo:TAG` is ONE file. + CHECK_FALSE(hub.asked_for("Q8_0")); + CHECK_FALSE(hub.asked_for("README")); +} + +TEST_CASE("model resolver: an unknown tag names the tags the listing HOLDS") { + FakeHub hub; + TempDir dir; + const HubEnvironment env(hub, dir.path() / "home"); + hub.add("Model-Q4_K_M.gguf", TinyGguf()); + hub.add("Model-Q8_0.gguf", TinyGguf()); + + ModelResolveOptions opts; + const std::string message = + MessageOf([&] { ResolveModelPath("org/repo:IQ2_XXS", opts); }); + INFO("refusal: " << message); + REQUIRE_FALSE(message.empty()); + CHECK(message.find("IQ2_XXS") != std::string::npos); + // A refusal that does not say what IS there costs the reader a second run. + CHECK(message.find("q4_k_m") != std::string::npos); + CHECK(message.find("q8_0") != std::string::npos); +} + +TEST_CASE("model resolver: --download-dir places the cache where it says") { + FakeHub hub; + TempDir dir; + const HubEnvironment env(hub, dir.path() / "home"); + PopulateSafetensorsRepo(hub); + + ModelResolveOptions opts; + opts.download_dir = dir.path() / "elsewhere"; + const std::string snapshot = ResolveModelPath("org/repo", opts); + CHECK(snapshot.rfind((dir.path() / "elsewhere").string(), 0) == 0); + CHECK(fs::is_regular_file(fs::path(snapshot) / "config.json")); + // And NOT under HF_HOME, which is what the flag exists to override. + CHECK_FALSE(fs::exists(dir.path() / "home" / "hub" / "models--org--repo")); +} + +TEST_CASE("model resolver: --revision names a branch, and there is no @rev syntax") { + FakeHub hub; + TempDir dir; + const HubEnvironment env(hub, dir.path() / "home"); + PopulateSafetensorsRepo(hub); + + ModelResolveOptions opts; + opts.revision = "main"; + CHECK_FALSE(ResolveModelPath("org/repo", opts).empty()); + + // `org/repo@abc` is NOT a form this row invented. vLLM spells the revision as + // its own flag (`arg_utils.py:839`), so the value falls through unchanged. + CHECK(ParseModelReference("org/repo@abc").kind == + ModelReference::kUnrecognized); + + ModelResolveOptions missing; + missing.revision = "no-such-branch"; + const std::string message = + MessageOf([&] { ResolveModelPath("org/repo", missing); }); + INFO("refusal: " << message); + CHECK(message.find("no-such-branch") != std::string::npos); +} + +TEST_CASE("model resolver: what an https endpoint does, on this build's TLS state") { + // BOTH ARMS ASSERT, for the reason recorded on the matching case in + // `test_downloader.cpp`: an `#ifndef` around the whole body turns this case + // into `assertions: 0` the moment W5 defines the macro, and a zero-assertion + // case is a skip wearing a pass. The endpoint is LOOPBACK on a port nothing + // listens on so that a TLS build fails to connect instead of reaching the + // network. + TempDir dir; + vllm_test::SetEnv("HF_ENDPOINT", "https://127.0.0.1:1/"); + vllm_test::SetEnv("HF_HOME", (dir.path() / "home").string().c_str()); + vllm_test::SetEnv("HF_HUB_OFFLINE", ""); + ModelResolveOptions opts; + const std::string message = + MessageOf([&] { ResolveModelPath("org/repo", opts); }); + vllm_test::SetEnv("HF_ENDPOINT", ""); + vllm_test::SetEnv("HF_HOME", ""); + INFO("refusal: " << message); + REQUIRE_FALSE(message.empty()); +#ifdef CPPHTTPLIB_OPENSSL_SUPPORT + // A build that speaks https reports the transport, not a rebuild instruction. + CHECK(message.find("cannot speak HTTPS") == std::string::npos); + CHECK(message.find("VLLM_CPP_OPENSSL") == std::string::npos); + CHECK(message.find("VLLM_CPP_BUILD_BORINGSSL") == std::string::npos); +#else + // Not a connection error that reads like a network fault. + CHECK(message.find("VLLM_CPP_HF_DOWNLOAD") != std::string::npos); + CHECK(message.find("VLLM_CPP_OPENSSL") != std::string::npos); + CHECK(message.find("VLLM_CPP_BUILD_BORINGSSL") != std::string::npos); +#endif +} + +TEST_CASE("model resolver: SelectWeightFiles is index driven, then format ordered") { + const std::vector listed = { + "config.json", "model-00001-of-00002.safetensors", + "model-00002-of-00002.safetensors", "pytorch_model.bin", + "original/model.safetensors"}; + + // INDEX WINS. The `weight_map` names exactly what to fetch, so the decoy and + // the second format never enter the answer. + const std::vector map = {"model-00001-of-00002.safetensors", + "model-00002-of-00002.safetensors"}; + CHECK(SelectWeightFiles(listed, map) == map); + + // With NO index, the FIRST matching pattern wins: safetensors before bin. + const std::vector globbed = SelectWeightFiles(listed, {}); + REQUIRE(globbed.size() == 2); + CHECK(globbed[0] == "model-00001-of-00002.safetensors"); + CHECK(globbed[1] == "model-00002-of-00002.safetensors"); + + // A repository with ONLY `.bin` gets `.bin`. + const std::vector bin_only = + SelectWeightFiles({"config.json", "pytorch_model.bin"}, {}); + REQUIRE(bin_only.size() == 1); + CHECK(bin_only[0] == "pytorch_model.bin"); + + // Nothing at the root means nothing to fetch, whatever a subdirectory holds. + CHECK(SelectWeightFiles({"original/model.safetensors"}, {}).empty()); + + CHECK(IsRepoRootFile("config.json")); + CHECK_FALSE(IsRepoRootFile("original/model.safetensors")); + CHECK_FALSE(IsRepoRootFile("")); +} + +// F6 of the fifth fresh review: the object-identifier PREFERENCE in +// `BlobNameFor` was unpinned. Dropping it left every case green, because both +// names are valid and nothing observable changed for a caller. What it does +// change is the cache file name, which is the thing `docs/USAGE.md` promises +// `--verbose` reports, so it is pinned here rather than recorded as owed. +TEST_CASE("model resolver: a blob is named by the object identifier, else by the commit and the path") { + FakeHub hub; + TempDir dir; + const fs::path home = dir.path() / "home"; + HubEnvironment env(hub, home); + // The identifier is only TRUSTED under a token, because an anonymous listing + // for a gated repository fabricates one. So the case that wants to see it + // used has to set one. + vllm_test::SetEnv("HF_TOKEN", "hf_a_test_token"); + + PopulateSafetensorsRepo(hub); + // Forty hexadecimal characters, not one repeated, so neither integrity rule + // fires on it. + const std::string kOid = "0123456789abcdef0123456789abcdef01234567"; + hub.add_oid("model.safetensors", kOid); + + const ModelResolveOptions opts; + const std::string resolved = ResolveModelPath("org/repo", opts); + vllm_test::SetEnv("HF_TOKEN", ""); + REQUIRE_FALSE(resolved.empty()); + + const fs::path blobs = home / "hub" / "models--org--repo" / "blobs"; + // The listing carried an identifier for the shard, so the identifier IS the + // blob name: it is a content hash, so two repositories holding one byte + // sequence share one blob. + CHECK(fs::is_regular_file(blobs / kOid)); + // It carried none for `config.json`, so the fallback is the commit and the + // flattened path, which names exactly one byte sequence for the same reason a + // content hash would. + CHECK(fs::is_regular_file(blobs / (std::string(kCommit) + "--config.json"))); + // And the identifier was not ALSO used as a second name for the same bytes. + CHECK_FALSE(fs::exists(blobs / + (std::string(kCommit) + "--model.safetensors"))); +}