Skip to content

nv_build provider unusable out of the box: default model is EOL, meta-analyzer slot model not served, bundled context length rejected #388

Description

@Mark2Mac

Title: nv_build provider is unusable out of the box: default model is EOL, meta-analyzer slot model is not served, and the bundled context length is rejected

Summary

With SKILLSPECTOR_PROVIDER=nv_build and no SKILLSPECTOR_MODEL, every LLM call fails.
Three separate problems in providers/nv_build/, all verifiable against
https://integrate.api.nvidia.com/v1/models and a single chat completion.

1. DEFAULT_MODEL is past end of life

# providers/nv_build/provider.py
DEFAULT_MODEL = "deepseek-ai/deepseek-v4-flash"
SLOT_DEFAULTS: dict[str, str] = {
    "meta_analyzer": "deepseek-ai/deepseek-v4-pro",
}
POST /v1/chat/completions  model=deepseek-ai/deepseek-v4-flash
410 {"type":"about:blank","title":"Gone","status":410,
     "detail":"The model 'deepseek-ai/deepseek-v4-flash' has reached its end of life
               on 2026-08-07T09:00:00Z and is no longer available"}

Neither deepseek-ai/deepseek-v4-flash nor deepseek-ai/deepseek-v4-pro appears in
GET /v1/models (102 models listed). The served successor is
deepseek-ai/deepseek-v4-flash-0731, which is not in the bundled registry.

So the default path and the meta-analyzer slot are both dead, and the resulting failure is an
opaque Connection error / Gone rather than "this model no longer exists".

2. z-ai/glm-5.1 in the registry is not served either

model_registry.yaml lists it at 205000 context; GET /v1/models does not return it.

3. The bundled context length for z-ai/glm-5.2 is rejected by the endpoint

model_registry.yaml declares:

"z-ai/glm-5.2":
  context_length: 1000000

with no max_output_tokens. model_info.get_max_output_tokens() then computes
ctx * (1 - MAX_INPUT_TOKENS_PCT) = 1_000_000 * 0.25 = 250 000, and every request fails:

400 {"message":"This model configuration accepts at most 202749 combined input and output
     tokens. However, your request has 1249 input tokens and asks for 250000 output tokens
     (251249 tokens total). Please reduce the input length or requested output length.",
     "type":"Bad Request","code":400}

Note the failure mode: an over-stated context window does not degrade gracefully, it
zeroes the LLM stage. Under-stating is safe; over-stating is not.

Impact

A new user following the documented nv_build path gets llm_degraded: true and a
static-only verdict, with nothing in the error text pointing at the model registry. On a
security scanner that is a silent loss of the semantic layer — the part that catches what
static analysis cannot. Measured on a bait skill with prose-disguised credential
exfiltration: static-only says LOW / SAFE, the LLM stage says CRITICAL / DO_NOT_INSTALL.

Suggested fix

  1. Point DEFAULT_MODEL and the meta_analyzer slot at models the catalogue currently
    serves (deepseek-ai/deepseek-v4-flash-0731 responds in ~1.6 s; z-ai/glm-5.2 in ~16 s).
  2. Give z-ai/glm-5.2 a max_output_tokens and a context length the endpoint accepts, or
    drop the entry so the conservative 128 000 default applies.
  3. Consider surfacing a 410/404 on the model as a distinct, named error. Today it reaches the
    user as Connection error. after a 60 s hang for some models
    (meta/llama-3.3-70b-instruct and openai/gpt-oss-120b are listed by /v1/models
    but close the connection without responding), which sends people looking at their network
    rather than at the model id.

Happy to send a PR for 1 and 2 if the shape is agreeable — the values are endpoint-observable,
though they may vary per account, which is itself worth a comment in the YAML.

Environment

SkillSpector 2.5.1 and main @ 2.9.5 (registry identical in both), Linux, Python 3.13,
free-tier account on build.nvidia.com.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions