Open OnDemand (OOD) Batch Connect application that starts llama-server in
an Apptainer container on a Slurm GPU node and exposes llama.cpp's bundled Web
UI through the OOD node proxy.
The application currently targets the tillicum cluster and supports the
gpu-h200 and gpu-h200-mig partitions.
- Proxy script path made portable:
before.sh.erbandscript.sh.erbnow default totemplate/proxy.pyinstead of a hard-coded home directory path.LLAMA_PROXY_PYremains an override. - Model catalogue aligned:
Google Gemma-4 (31B-it-UD-IQ3_XXS)enabled forgpu-h200-miginform.yml;Meta Muse-Glimmeradded to documentation; duplicate placeholder removed. view.html.erbcsrftoken cookie is now set once on page load; the duplicateonclicksetter was removed.- Documentation updated to reflect current file paths and model list.
| Setting | Current value |
|---|---|
| Cluster | tillicum |
| QoS | normal |
| GPUs | exactly 1 |
| Duration | 4 hours |
| Default partition | gpu-h200 |
| Container | /gpfs/containers/llama.cpp/llama.cpp_full-cuda13.sif |
| Temperature | 0.8 |
| Context size | model maximum (--ctx-size 0) |
| Built-in llama.cpp tools | enabled (--tools all) |
The number of GPUs and container path are hidden, administrator-managed form values. The generated Slurm request always asks for one GPU.
Browser
│
▼
OOD /rnode/<compute-host>/<ood-port>/
│
▼
Python reverse proxy on 0.0.0.0:<ood-port>
│ strips the OOD prefix and normalizes request headers
▼
llama-server on 127.0.0.1:<inference-port>
│
▼
Apptainer container + allocated GPU
Two different random ports are allocated:
portis the externally routed OOD node-proxy port.llama_portis the loopback-only llama.cpp inference port.
The proxy starts immediately. Until llama-server finishes loading the GGUF,
it returns a friendly auto-refreshing loading page instead of a raw upstream
error.
| Path | Purpose |
|---|---|
manifest.yml |
OOD name, menu category, role, and description |
form.yml |
Form fields, model catalog, defaults, and dynamic OOD directives |
form.js |
Cost estimate, advanced-field visibility, and partition/model filtering |
submit.yml.erb |
Slurm resources and optional extra Slurm arguments |
template/before.sh.erb |
Port allocation and validation of the container, model, Python, and proxy |
template/script.sh.erb |
Starts llama-server, the proxy, watchdog, and cleanup logic |
template/after.sh |
Waits up to 120 seconds for the proxy port |
template/proxy.py |
Standard-library Python reverse proxy and WebUI subpath shim |
view.html.erb |
OOD session page and Launch Chat button |
Resources are selected from gpu_partition in submit.yml.erb:
| Partition | GPUs | CPUs per task | Memory | Estimated GPU rate |
|---|---|---|---|---|
gpu-h200 |
1 | 8 | 240 GB | $0.90/hour |
gpu-h200-mig |
1 MIG GPU | 2 | 34 GB | $0.13/hour |
The 34 GB MIG request is intentional: the scheduler rejects requests above 34 GB per requested MIG GPU.
bc_num_hours is converted to minutes for Slurm. For example, 4 hours becomes:
--time=240:00
The cost shown by form.js is an estimate only:
1 GPU × requested hours × partition rate
The selected account and QoS come from OOD's auto_accounts and auto_qos
fields. OOD may restore a user's cached form selection, so the value visible in
the browser can differ from the declared normal QoS default.
form.js filters the Model dropdown whenever the partition changes. Each
model option also has an OOD data-option-for-... directive, so native OOD
dynamic forms and the application JavaScript express the same relationship.
The JavaScript fallback remains necessary on sites where bc_dynamic_js is
disabled.
| Partition | Display name | GGUF path | Status |
|---|---|---|---|
gpu-h200 |
Alibaba Qwen3.8 (27B-bf16) | /gpfs/models/Qwen3.8-27B.gguf |
enabled |
gpu-h200 |
NVIDIA Nemotron-3.5-Lightning (30B-A3B-bf16) | /gpfs/models/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf |
enabled |
gpu-h200 |
Google Gemma-4 (31B-it-bf16) | /gpfs/models/gemma-4-31B-it-GGUF/BF16/gemma-4-31B-it-BF16-00001-of-00002.gguf |
enabled |
gpu-h200 |
Meta Muse-Glimmer (30B-fp4) | /gpfs/models/Muse-Glimmer-30B-GGUF/Muse-Glimmer-30B-KQuant-Dynamic-Q4_K_XL.gguf |
enabled |
gpu-h200-mig |
OpenBMB MiniCPM (2B-fp16) | /gpfs/models/MiniCPM5-2B-GGUF/MiniCPM5-2B-F16.gguf |
enabled |
gpu-h200-mig |
Alibaba Qwen3.5 (2B-bf16) | /gpfs/models/Qwen3.5-2B.gguf |
enabled |
gpu-h200-mig |
Google Gemma-4 (31B-it-UD-IQ3_XXS) | /gpfs/models/gemma-4-31B-it-GGUF/gemma-4-31B-it-UD-IQ3_XXS.gguf |
enabled |
gpu-h200-mig |
Alibaba Qwen3.5 (4B-bf16) | /gpfs/models/Qwen3.5-4B.gguf |
disabled placeholder |
gpu-h200-mig |
Alibaba Qwen3.5 (9B-bf16) | /gpfs/models/Qwen3.5-9B.gguf |
disabled placeholder |
The model_path YAML default is intentionally empty. On page initialization,
form.js selects the first enabled model compatible with the current
partition. If a partition has no enabled models, it clears the selection,
marks the required field invalid, and displays a warning.
The model_path YAML default is intentionally empty. On page initialization,
form.js selects the first enabled model compatible with the current
partition. If a partition has no enabled models, it clears the selection,
marks the required field invalid, and displays a warning.
Add an option under attributes.model_path.options in form.yml.
Full H200 example:
- ["Model name", "/gpfs/models/model.gguf",
{data-partition: "gpu-h200",
data-option-for-gpu-partition-gpu-h200-mig: false}]MIG example:
- ["Model name", "/gpfs/models/model.gguf",
{data-partition: "gpu-h200-mig",
data-option-for-gpu-partition-gpu-h200: false}]For a non-selectable future placeholder, add these attributes to the option:
disabled: true
data-model-placeholder: trueKeep all option HTML attributes in one hash in the third array position. The model path must be readable from compute nodes and the model must fit within the selected partition's GPU allocation.
Any model whose path is under /gpfs/models/gemma-4-31B-it-GGUF/ also receives:
--mmproj /gpfs/models/gemma-4-31B-it-GGUF/mmproj-BF16.gguf
The script bind-mounts both the model directory and the projector directory into the container.
The Explore advanced options? checkbox controls these visible fields:
| Field | Effective behavior |
|---|---|
| Temperature | Uses the selected value when advanced options are enabled; otherwise forced to 0.8 |
| Extra llama-server arguments | Passed only when advanced options are enabled |
| Extra Slurm command-line options | Added only when advanced options are enabled |
api_key is currently a hidden field with an empty administrator-defined
value; users cannot edit it in the launch form. If configured, it is encoded
during ERB rendering and supplied to llama-server as --api-key.
The current form value for extra llama-server arguments is:
--jinja --webui-mcp-proxy
Because advanced arguments are gated, those flags are passed only after the
checkbox is selected. In the currently deployed llama.cpp container, Jinja is
enabled by default; --webui-mcp-proxy is needed only when using the WebUI MCP
CORS proxy.
Extra argument strings are split on whitespace. Shell-style quoting inside the form value is not preserved, so this field is best suited to simple flags and single-token values.
- Define the attribute in
form.ymland add it to theformlist. - Add its underscore-form name to
ADVANCED_FIELDSinform.js. - Under
enable_advanced_args.html_options.data, add an OOD directive using hyphens instead of underscores:
hide-context-size-when-un-checked: trueHiding a field does not clear its submitted value. If an option must have no
effect while advanced settings are disabled, gate it in submit.yml.erb or
template/script.sh.erb, as is currently done for temperature and extra
arguments.
The effective command is equivalent to:
apptainer run --nv \
--bind "$(dirname "$MODEL_PATH")" \
"$LLAMA_SIF" serve \
-m "$MODEL_PATH" \
--host 127.0.0.1 \
--port "$LLAMA_PORT" \
--threads "$SLURM_CPUS_PER_TASK" \
--ctx-size 0 \
--temp "$TEMPERATURE" \
--tools all \
[--mmproj ...] \
[--api-key ...] \
[extra llama-server arguments]--tools all enables the built-in tools supplied by this llama.cpp build. It
does not inherently give the model a dedicated web-search or URL-fetch tool.
Tool availability also depends on the WebUI settings, model chat template, and
user approval. Enabling all tools can expose filesystem and shell operations on
the compute node; treat this application as trusted-user functionality.
template/proxy.py is a threaded, standard-library-only HTTP proxy. It:
- strips
/rnode/<host>/<port>before forwarding requests; - rewrites
Host,Origin, andRefererfor llama.cpp's localhost CORS policy; - injects a small JavaScript shim into HTML before the WebUI scripts run;
- prefixes root-absolute WebUI endpoints such as
/v1/models,/models/*,/tools,/mcp-servers, and/cors-proxy; - patches root-absolute module-preload links;
- disables service-worker registration to avoid stale per-port PWA caches;
- streams SSE, downloads, and compressed assets without buffering; and
- returns an auto-refreshing loading page while the upstream server is not ready.
The container's WebUI assets are pre-compressed. The proxy preserves the
browser's Accept-Encoding header and passes non-HTML gzip responses through
unchanged. It decompresses HTML only when necessary to inject the subpath shim.
Streaming HTML patching was evaluated. Full streaming decompression with
</head> boundary detection across gzip/deflate chunks introduces complexity
and edge-case failures for partial tags. The current buffered approach for HTML
is retained for reliability; non-HTML responses remain streamed.
A development app can be linked into the OOD development app directory. A production deployment is normally cloned or copied into the site's system app root according to local OOD administration policy.
The proxy source is referenced relative to the app root:
template/proxy.py
Both before.sh.erb and script.sh.erb default to template/proxy.py and can be
overridden with the LLAMA_PROXY_PY environment variable. This removes the
previous hard-coded /gpfs/home/npho/ondemand/dev/llama-webui/... path and
makes the app portable.
The manifest places the application under:
Interactive Apps → LLM Tools → Llama.cpp WebUI
OOD displays the root VERSION file when one exists. Otherwise it uses:
git describe --always --tagsTagging a release therefore produces a friendlier version label:
git tag -a 1.0 -m "Llama WebUI release 1.0"
git push origin 1.0Without a VERSION file or matching tag, OOD displays an abbreviated commit
hash.
Before launch, the application checks that:
- the Apptainer image is readable;
- the selected GGUF is readable;
python3is available; and- the proxy script is readable.
OOD's primary session output is stored beneath its Batch Connect session output directory. Runtime component logs are written on the compute node to:
${TMPDIR:-/tmp}/llama-webui-${SLURM_JOB_ID}/llama-server.log
${TMPDIR:-/tmp}/llama-webui-${SLURM_JOB_ID}/proxy.log
Set PROXY_DEBUG=1 in the job environment for verbose proxy request logging.
The watchdog prints the final 30 llama-server log lines to OOD output if the
server process exits unexpectedly.
After submitting a session:
- Confirm the session reaches Running and then Connected.
- Click Launch Chat.
- Expect the loading page while the model initializes.
- Confirm the llama.cpp WebUI appears and lists the selected model.
- Send a prompt and verify that token streaming works.
- If using tools, confirm they appear in the WebUI and test a harmless, read-only tool first.
Confirm the generated request uses:
--gpus 1 --cpus-per-task=2 --mem=34G --partition=gpu-h200-mig
Requests above 34 GB per MIG GPU are rejected by the current scheduler policy. Also check account, QoS, reservation, and node availability in the scheduler's actual error message.
Hard-refresh the OOD launch form. Inspect each model option's data-partition
and data-option-for-... attributes. Browser-cached form values can be
restored by OOD, but form.js should replace an incompatible selection during
initialization.
Close or hard-refresh an already-open launch page before resubmitting. OOD may
also restore cacheable user selections. Inspect the session's
user_defined_context.json and job_script_options.json to see the exact form
values and Slurm request used for that launch.
The request reached llama.cpp without an acceptable gzip encoding. Verify that
the deployed proxy.py preserves Accept-Encoding, then start a new session
so the updated proxy is copied into the session directory.
Check the browser developer console and proxy.log. Root-absolute API requests
must be rewritten beneath /rnode/<host>/<port>. A stale proxy or WebUI change
that introduces a new root-absolute endpoint may require updating the shim.
Inspect llama-server.log for an unreadable GGUF, invalid projector, GPU
visibility failure, unsupported model architecture, or out-of-memory error.
The proxy can be healthy and display the loading page even when llama-server
has failed; the watchdog should then terminate the proxy and copy the final
server log lines into OOD output.
Both ports are initially selected with OOD's find_port. The inference port is
rechecked before startup and falls back to scanning ports 15000–15099 if
needed.
The WebUI is embedded in the llama.cpp container. Updating this OOD repository does not update that UI; rebuild or replace the SIF and launch a new session.