Skip to content

chore: trigger NVSkills CI for cuopt-numerical-optimization-api #1674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rapids-bot merged 5 commits into from
Aug 7, 2026
Merged

chore: trigger NVSkills CI for cuopt-numerical-optimization-api #1674

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
dd7e77f
chore: trigger NVSkills CI for cuopt-numerical-optimization-api
ramakrishnap-nv Aug 5, 2026
b064bbb
Merge branch 'main' into chore/nvskills-ci-cuopt-numerical-optimizati…
ramakrishnap-nv Aug 5, 2026
25706dd
Attach NVSkills validation signatures
svc-nvskills-signing Aug 5, 2026
41eda3f
Merge branch 'main' into chore/nvskills-ci-cuopt-numerical-optimizati…
ramakrishnap-nv Aug 6, 2026
78e8627
Merge branch 'main' into chore/nvskills-ci-cuopt-numerical-optimizati…
ramakrishnap-nv Aug 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 73 additions & 53 deletions skills/cuopt-numerical-optimization-api/BENCHMARK.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,81 +1,101 @@
# Evaluation Report
# Skill Benchmark: cuopt-numerical-optimization-api

Evaluation of the `cuopt-numerical-optimization-api` skill before publication through NVSkills-Eval.
> ✅ **Overall verdict: PASS — Recommended for publication**

This benchmark summarizes 3-Tier Evaluation from NVSkills-Eval results for the skill. The goal is to document whether the skill is safe, discoverable, effective, and useful for agents before it is published for broader workflow use.
## Publication Recommendation

Recommended for publication based on the completed evaluation evidence in this report.

## Evaluation Summary
## Evaluation Metadata

- Skill: `cuopt-numerical-optimization-api`
- Evaluation date: 2026-07-02
- NVSkills-Eval profile: `external`
- Environment: `astra-sandbox`
- Dataset: 9 evaluation tasks
- Evaluation date: 2026-08-05
- Evaluator version: `1.0.0`
- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)
- Tasks: 9 evaluation tasks (9 positive)
- Dataset digest: `sha256:f385c69ce235035e8e5cbfd91a8d3245b75473f0bf0d694451e988e439f6e9e1` (skill-evaluator-dataset-snapshot/1)
- Attempts per task: 1
- Pass threshold: 50%
- Overall verdict: PASS
- Environment: `k8s-sandbox`
- Tier 3 evidence: required for publication

## Agents Used
Each task attempt ran in its own isolated sandbox pod.

- `claude-code`
- `codex`
## What This Report Answers

## Metrics Used
The three-tier evaluation checks whether the skill:

Reported benchmark dimensions:
- is safe to use;
- produces correct answers;
- is discovered and activated when needed;
- helps the agent complete the user's goal and expected workflow; and
- avoids wasted skill and tool usage.

- Security: checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access.
- Correctness: checks whether the agent follows the expected workflow and produces the correct final output.
- Discoverability: checks whether the agent loads the skill when relevant and avoids using it when irrelevant.
- Effectiveness: checks whether the agent performs measurably better with the skill than without it.
- Efficiency: checks whether the agent uses fewer tokens and avoids redundant work.
## Results at a Glance

Underlying evaluation signals used in this run:
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
|---|---:|---:|
| Overall | 59% → 97% (+39 points) | 61% → 93% (+32 points) |
| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) |
| Correctness | 87% → 98% (+11 points) | 91% → 100% (+9 points) |
| Discoverability | 28% → 100% (+72 points) | 44% → 92% (+48 points) |
| Effectiveness | 69% → 89% (+20 points) | 68% → 81% (+12 points) |
| Efficiency | 10% → 100% (+90 points) | 0% → 90% (+90 points) |

- `security` (Security): checks for unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): verifies that the agent loaded the expected skill and workflow.
- `skill_efficiency` (Efficiency): checks routing quality, decoy avoidance, and redundant tool usage.
- `accuracy` (Accuracy): grades final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): checks whether the overall user task completed successfully.
- `behavior_check` (Behavior Check): verifies expected behavior steps, including safety expectations.
- `token_efficiency` (Token Efficiency): compares token usage with and without the skill.
**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points.

## Test Tasks
Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline.
Comment on lines +35 to +46

@coderabbitai coderabbitai Bot Aug 5, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -eu

for file in skills/cuopt-numerical-optimization-api/BENCHMARK.md \
            skills/cuopt-numerical-optimization-api/skill-card.md; do
  echo "===== $file ====="
  sed -n '25,55p' "$file" 2>/dev/null || true
  sed -n '60,90p' "$file" 2>/dev/null || true
done

echo "===== related metric definitions and source values ====="
rg -n -i -C 3 'baseline|uplift|overall|effectiveness|raw score|rounding|percentage points|benchmark' \
  skills/cuopt-numerical-optimization-api

Repository: NVIDIA/cuopt

Length of output: 1947

🌐 Web query:

site:github.com/NVIDIA/cuopt "Claude Code (Baseline → Skill Uplift)" "Efficiency"

💡 Result:

Claude Code is a tool-use capable AI coding agent utilized in software development environments to automate tasks such as code generation, refactoring, issue resolution, and documentation [1][2][3][4]. In the context of "baseline to skill uplift" efficiency, this refers to the process of enhancing an AI agent's performance by moving from a general-purpose baseline capability to using specialized, curated "skills" [5][4]. NVIDIA's cuOpt project provides a practical example of this approach. Developers use Claude Code to perform tasks, and performance is "uplifted" by equipping the agent with specific domain-knowledge packages called "skills" [5][4]. Key aspects of this efficiency model include: 1. Baseline Capability: The AI agent begins with general coding knowledge, which is sufficient for standard tasks but may struggle with highly specialized or complex, multi-objective problem formulations [5]. 2. Skill Uplift: Developers introduce "skills"—which may include domain-specific APIs, structured documentation, and guided workflows—that the agent can reference to perform tasks more accurately and efficiently [5][4]. For example, a "multi-objective exploration" skill allows an agent to move beyond single-objective optimization to exploring Pareto frontiers, providing better guidance on trade-offs rather than just returning a single answer [5]. 3. Performance Metrics: Efficiency gains are often measured through A/B testing on synthetic instances or standard benchmarks [5]. This involves comparing the agent's output "with vs. without" the specific skill to ensure the agent demonstrates better adherence to requirements, improved logical reasoning, or faster task completion [5]. 4. Agentic Evolution: The framework often includes a "skill-evolution" workflow where the skills themselves are maintained, evaluated, and updated over time, allowing the agent to continuously improve its performance on domain-specific tasks without needing a fundamental retraining of the underlying model [4]. This structure allows technical teams to standardize agent behavior, reduce hallucinations or errors in specialized domains, and improve the reliability of AI-generated code in production-critical environments [5][4].

Citations:

🌐 Web query:

site:github.com/NVIDIA/cuopt "How to read this table" "skill score - baseline score"

💡 Result:

In technical contexts, particularly within performance evaluation, model assessment, or skill-based frameworks (such as the NVIDIA cuOpt Agent Skills system), comparing a skill score to a baseline score is a method for measuring the value-add of a specific model, agent, or intervention over a reference point [1][2][3]. How to interpret this comparison: 1. Skill Score: This represents the performance metric achieved by your target system (e.g., an AI agent, a specific configuration, or an optimized model) under test [1][3]. It reflects the actual capability or outcome of the model being evaluated [4][3]. 2. Baseline Score: This is the reference performance metric, representing a standard, default, or "naive" approach against which the target system is compared [2][3]. It establishes a floor for performance, showing what could be achieved without the specific skill or improvement being tested [5][3]. 3. Interpretation: - If the Skill Score > Baseline Score: The target system provides a measurable improvement, demonstrating that the skill or modification adds value [1]. - If the Skill Score = Baseline Score: The system is performing at the same level as the standard or naive approach, suggesting the skill or modification is ineffective for that specific metric. - If the Skill Score < Baseline Score: The modification is negatively impacting performance, indicating that the target system performs worse than the baseline [1]. In systems like the cuOpt agent framework, these scores are used to validate "skill" updates—where an agent iterates on a task, calculates a score, and compares it against previous iterations (baseline) to decide whether to distill that learning into a permanent improvement [2][3]. A higher score relative to the baseline confirms that the new logic or configuration is a beneficial contribution to the agent's capabilities [1][3].

Citations:

🏁 Script executed:

#!/bin/bash
set -eu

base="https://raw.githubusercontent.com/NVIDIA/cuopt/main"
for path in \
  skills/cuopt-numerical-optimization-api/BENCHMARK.md \
  skills/cuopt-numerical-optimization-api/skill-card.md; do
  echo "===== $path ====="
  curl --fail --silent --show-error "$base/$path" \
    | nl -ba \
    | sed -n '25,90p'
done

Repository: NVIDIA/cuopt

Length of output: 1947

Correct the uplift values in both tables.

The displayed scores yield Claude Code Overall +38 (97 - 59) and Codex Effectiveness +13 (81 - 68). If unrounded scores produce different uplifts, document the rounding rule.

📍 Affects 2 files
  • skills/cuopt-numerical-optimization-api/BENCHMARK.md#L35-L46 (this comment)
  • skills/cuopt-numerical-optimization-api/skill-card.md#L70-L77
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/cuopt-numerical-optimization-api/BENCHMARK.md` around lines 35 - 46,
Correct the uplift values in both
skills/cuopt-numerical-optimization-api/BENCHMARK.md lines 35-46 and
skills/cuopt-numerical-optimization-api/skill-card.md lines 70-77 so they match
the displayed rounded scores, including Claude Code Overall as +38 and Codex
Effectiveness as +13. If the values intentionally use unrounded scores, document
the rounding rule in both tables.


The benchmark dataset contained 9 evaluation tasks:
## Tier Status

- Positive tasks: 9 tasks where the skill was expected to activate.
- Negative tasks: 0 tasks where no skill was expected.
- Unlabeled tasks: 0 tasks where positive/negative intent could not be inferred.
| Tier | Purpose | Status | Evidence |
|---|---|---|---|
| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 4 finding(s) |
| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded |
| Tier 3 | Live agent evaluation | **PASS** | 2 agent(s); 9 task(s) |

Task composition is derived from the evaluation dataset when possible. Entries with `expected_skill` set are treated as positive skill-activation cases, while entries with `expected_skill: null` are treated as negative activation cases.
## Findings and Observations

## Results
<details>
<summary>Show detailed findings and successful checks</summary>

| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 8 | 100% (+0%) | 100% (+0%) |
| Correctness | 8 | 76% (+33%) | 70% (+12%) |
| Discoverability | 8 | 67% (+58%) | 51% (+31%) |
| Effectiveness | 8 | 73% (+20%) | 69% (+7%) |
| Efficiency | 8 | 71% (+46%) | 55% (+28%) |
- **MEDIUM** SCHEMA/frontmatter_field_placement: Root field 'version' is ignored; use 'metadata.version' (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- **LOW** SCHEMA/author_format: Author must be of the form 'Name <email@host>' (`skills/cuopt-numerical-optimization-api/SKILL.md`)

Score values show skill-assisted performance. Values in parentheses show uplift versus the no-skill baseline when baseline data is available.
</details>

## Tier 1: Static Validation Summary
## Scoring Methodology

Tier 1 validation passed with observations. NVSkills-Eval ran 1 checks and found 3 total findings.
<details>
<summary>Show dimension definitions, source signals, and thresholds</summary>

Top findings:
| Dimension | Question | Scored signals |
|---|---|---|
| Security | Is it safe to use? | `security` (100%) |
| Correctness | Is the answer correct? | `accuracy` (100%) |
| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) |

- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Instructions' (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- MEDIUM SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- LOW SCHEMA/author_format: Author must be of the form 'Name <email@host>' (`skills/cuopt-numerical-optimization-api/SKILL.md`)
- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.

## Tier 2: Deduplication Summary
Signals present in this run:

This tier was not run or did not produce findings in this report.
- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
- `skill_execution` (Skill Execution): whether the expected skill was found and executed.
- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use.
- `accuracy` (Accuracy): final-answer correctness against the reference answer.
- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.

## Publication Recommendation
</details>

## Freshness

The skill is suitable to proceed toward NVSkills-Eval publication based on this benchmark. Skill owners should keep this file with the skill and refresh it when the evaluation dataset, skill behavior, or target agents materially change.
Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.
1 change: 1 addition & 0 deletions skills/cuopt-numerical-optimization-api/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ metadata:
---



# cuOpt Numerical Optimization API

Model and solve LP, MILP, and QP problems using NVIDIA cuOpt's GPU-accelerated solver.
Expand Down
61 changes: 33 additions & 28 deletions skills/cuopt-numerical-optimization-api/skill-card.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,22 @@
## Description: <br>
Model and solve LP, MILP, and QP (beta) problems using NVIDIA cuOpt's GPU-accelerated solver across Python, C, and CLI interfaces. <br>
Model and solve LP, MILP, and QP problems using NVIDIA cuOpt's GPU-accelerated solver via Python, C, and CLI interfaces. <br>

This skill is ready for commercial/non-commercial use. <br>

## Owner
NVIDIA <br>

### License/Terms of Use: <br>
Apache-2.0 <br>
Apache 2.0 <br>
## Use Case: <br>
Developers and engineers solving linear programming, mixed-integer linear programming, and quadratic programming problems using NVIDIA cuOpt's Python, C, or CLI interfaces. <br>
Developers and engineers solving linear programming, mixed-integer linear programming, and quadratic programming optimization problems using NVIDIA cuOpt across Python, C, and CLI interfaces. <br>

### Deployment Geography for Use: <br>
Global <br>

## Requirements / Dependencies: <br>
**Requires API Key or External Credential:** [No] <br>
**Credential Type(s):** [None] <br>
**Requires API Key or External Credential:** [Not Specified] <br>
**Credential Type(s):** [None identified] <br>

Do not include secrets in prompts/logs/output; use least-privilege credentials; rotate keys as appropriate. <br>

Expand All @@ -34,47 +34,52 @@ Mitigation: Review and scan skill before deployment. <br>


## Skill Output: <br>
**Output Type(s):** [Code, Shell commands, Configuration instructions] <br>
**Output Format:** [Markdown with inline code blocks (Python, C, bash)] <br>
**Output Type(s):** [Code, Shell commands, Configuration instructions, Analysis] <br>
**Output Format:** [Markdown with inline code blocks] <br>
**Output Parameters:** [1D] <br>
**Other Properties Related to Output:** [None] <br>

## Evaluation Agents Used: <br>
- claude-code <br>
- codex <br>
- Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`) <br>
- Codex (`openai/openai/gpt-5.5`) <br>



## Evaluation Tasks: <br>
Evaluated against 9 internal evaluation tasks (NVSkills-Eval external profile, astra-sandbox environment). <br>
9 evaluation tasks (9 positive) in isolated sandbox pods. <br>

## Evaluation Metrics Used: <br>
Reported benchmark dimensions: <br>
- Security: Checks whether skill-assisted execution avoids unsafe behavior such as secret leakage, destructive commands, or unauthorized access. <br>
- Correctness: Checks whether the agent follows the expected workflow and produces the correct final output. <br>
- Discoverability: Checks whether the agent loads the skill when relevant and avoids using it when irrelevant. <br>
- Effectiveness: Checks whether the agent performs measurably better with the skill than without it. <br>
- Efficiency: Checks whether the agent uses fewer tokens and avoids redundant work. <br>
- Security: Whether the skill is safe to use (unsafe operations, secret leakage, unauthorized access). <br>
- Correctness: Whether the final answer is correct against the reference answer. <br>
- Discoverability: Whether the right skill was found and executed when needed. <br>
- Effectiveness: Whether the skill helps complete the user's goal and expected workflow. <br>
- Efficiency: Whether the skill avoids wasted tool or skill usage. <br>

Underlying evaluation signals used in this run: <br>
- `security`: Checks for unsafe operations, secret leakage, and unauthorized access. <br>
- `skill_execution`: Verifies that the agent loaded the expected skill and workflow. <br>
- `skill_efficiency`: Checks routing quality, decoy avoidance, and redundant tool usage. <br>
- `accuracy`: Grades final-answer correctness against the reference answer. <br>
- `goal_accuracy`: Checks whether the overall user task completed successfully. <br>
- `behavior_check`: Verifies expected behavior steps, including safety expectations. <br>
- `token_efficiency`: Compares token usage with and without the skill. <br>
- `skill_execution`: Whether the expected skill was found and executed. <br>
- `skill_efficiency`: Routing quality, workspace-aware skill reads, and productive tool use. <br>
- `accuracy`: Final-answer correctness against the reference answer. <br>
- `goal_accuracy`: Whether the user's goal was achieved. <br>
- `behavior_check`: Whether the expected workflow behavior was followed. <br>



## Evaluation Results: <br>
| Dimension | Num | `claude-code` | `codex` |
|---|---:|---:|---:|
| Security | 8 | 100% (+0%) | 100% (+0%) |
| Correctness | 8 | 76% (+33%) | 70% (+12%) |
| Discoverability | 8 | 67% (+58%) | 51% (+31%) |
| Effectiveness | 8 | 73% (+20%) | 69% (+7%) |
| Efficiency | 8 | 71% (+46%) | 55% (+28%) |
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
|---|---:|---:|
| Overall | 59% → 97% (+39 points) | 61% → 93% (+32 points) |
| Security | 100% → 100% (±0 points) | 100% → 100% (±0 points) |
| Correctness | 87% → 98% (+11 points) | 91% → 100% (+9 points) |
| Discoverability | 28% → 100% (+72 points) | 44% → 92% (+48 points) |
| Effectiveness | 69% → 89% (+20 points) | 68% → 81% (+12 points) |
| Efficiency | 10% → 100% (+90 points) | 0% → 90% (+90 points) |

## Testing Completed: <br>
**[x] Agent Red-Teaming** <br>
**[ ] Network Security** <br>
**[ ] Product Security** <br>

## Skill Version(s): <br>
26.10.00 (source: frontmatter) <br>
Expand Down
Loading