Ship a Claude Code skill and an MCP server with the CLI

[arav-agarwal2]

Summary

Two ways for Claude to drive the CLI, both shipped in the package.

Claude Code skill

• skills/mlperf-submissions/SKILL.md ships as package data.
• endpoints-submission-cli install-skill copies it to ~/.claude/skills/. Use --project for ./.claude/skills/, or --dest for anywhere else.
• It's versioned with the CLI. Re-running is a no-op when the copy is identical, and an edited copy is only replaced with --force.
• It teaches Claude the commands and their flags, to use --json, and to confirm every write with the user first.

MCP server

• endpoints-submission-mcp, installed by the new mcp extra:

  claude mcp add mlperf -e PRISM_USER_API_TOKEN=mlc_... \
    -- uvx --from 'endpoints-submission-cli[mcp]' endpoints-submission-mcp
  

• Every command is a tool, 16 in all. Each runs the installed CLI under the server's own interpreter. Commands with --json return parsed JSON; the rest return their status text.
• Get and download are split. runs get and submissions get each become a get tool and a download tool, because --download-to appends text after the JSON.
• Annotations: 5 read-only and 4 destructive (update_submission, remove_run_from_submission, withdraw_submission, delete_run). Clients use these to decide what to confirm with the user.
• Provisional submissions need confirm_public_provisional=true, which stands in for the CLI's interactive prompt. The server closes the command's input, so any prompt fails at once instead of hanging.
• Auth comes from the server's environment, so no tool takes a token.

Packaging

• New mcp extra pinned to mcp>=2,<3. 2.x renamed FastMCP to MCPServer, and the server is written against 2.x.
• Second console script (endpoints-submission-mcp) and a package-data entry for the skill.
• __main__.py, so the server can run the CLI with its own interpreter (python -m endpoints_submission_cli).
• mcp is also in the dev extra, so CI type-checks and tests the server.
• Most of the diff is uv.lock: mcp and its dependencies.

Notes for review

• Error messages: mcp 2.x hides the message of any exception other than ToolError. Without the conversion, "No API token provided" reached the client only as "Error executing tool".
• Options not exposed: --token, and check-submission's --quiet and -o. The tool already returns the full report.
• Not checked live: no tool that writes to PRISM was called against the live API. With a test account, create_run(..., test=True) would cover that end to end.

Test plan

• pytest: 1244 passed. ruff and mypy clean.
• A new test fails if the CLI gains a command or an option that no tool reaches. Confirmed by deleting --pinned from create_run, which failed with runs create: no tool sets {'pinned'}.
• The commands the skill names exist in the CLI.
• The built wheel contains the server, the skill, both scripts, and the mcp extra as an extra, not a hard dependency.
• Fresh environment without the extra: endpoints-submission-mcp says to install [mcp], and install-skill works.
• Real MCP client over stdio:
  • 16 tools listed with the annotations above.
  • check_submission passes valid_standardized and reports 17 errors on invalid_submission.
  • create_run with dry_run returns the payload.
  • A provisional submission without the confirmation is refused.
  • With no token, the CLI's auth message reaches the client.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[arav-agarwal2]

Double-checked by hand - this LGTM.

This should help a fair bit in terms of UI/UX

[anandhu-eng]

LGTM