Start with the project README to build NInfer, download a published artifact, and run the CLI or HTTP server.
| Document | Purpose |
|---|---|
| CLI | text, chat-history, image/video input, output streams, sampling, MTP, and common runtime options |
| HTTP serving | OpenAI Responses/Chat Completions, Anthropic Messages, state, streaming, token counting, authentication, and tool calls |
| Performance | RTX 5090 measurement coverage, per-model serving results, methodology, and publication rules |
| Weight conversion | official recipes, custom formats and sources, conversion methods, optional components and artifact output |
| Perplexity | fixed-corpus and custom-text causal perplexity, comparison rules, progress, and reports |
| CLI examples | committed text, multimodal, thinking, long-decode, and long-context inputs |
The executable --help output is the exact source for command-line option spelling and defaults.
| Model | Weights | Download | Versioned model card source |
|---|---|---|---|
| Qwen3.6-27B | groupwise-int |
Hugging Face | model card |
| Qwen3.6-27B | nvfp4 |
Hugging Face | model card |
| Qwen3.8-27B | groupwise-int |
Hugging Face | model card |
| Qwen3.8-27B | nvfp4 |
Hugging Face | model card |
| Qwen3.6-35B-A3B | groupwise-int |
Hugging Face | model card |
The active references under maintainer/ record current architecture, model,
artifact, and maintenance contracts. These files are not additional user workflows or installed
API documentation.
Engine architecture is the single top-level reference. The other references own narrower contracts:
| Document | Responsibility |
|---|---|
| Engine architecture | model/config/weight ownership, loading-to-execution flow, requests, scheduling, transactions and graphs |
| Build system | CMake targets, explicit source ownership, CUDA compilation boundaries, presets and developer configuration |
| Artifact container | v3 directory, objects, logical bindings, Uses, resources and file framing/sharding |
| Numeric formats | represented values, codes/scales, conversion arithmetic and numerical interpretation |
| Storage layouts | packing, plane offsets, padding, encoded sizes and view addressing |
| Qwen3.5 model | Dense/MoE mathematics, instance config, logical parameters, MTP, Vision and state semantics |
| DFlash and DFlash2 | conditioning, masked draft computation, proposal distributions and backend state |
| Resource scheduling and context cache | candidate selection, retention, materialization and Device/Host checkpoint policy |
| Paged KV context store | typed pools, pages, replicas, address spaces, reservations and consumer views |
| ReplaySSM GDN | raw transition records and faithful commitment of the verified state prefix |
| Op development | semantic boundaries, source ownership, numerical qualification and performance evidence |
| Operational logging | log ownership, presentation, severity and data policy |
| Linear benchmark | pure Linear measurement, metrics and suites |
| Linear tuning and reports | tuning ranges, priority points, dispatch tradeoffs and final performance report format |
Model cards contain official artifact facts and source provenance. The conversion guide is the entry point for making an artifact. Exact config fields, parameter expansion and native supported domains are maintained by the code linked from these references.