kubectl plugin for managing NodeWright deployments, packages, and nodes.
The NodeWright CLI (kubectl nodewright) provides SRE tooling for managing NodeWright deployments and their packages across Kubernetes cluster nodes. It supports inspecting node/package state, forcing re-runs, managing node lifecycle, and retrieving logs.
This CLI targets the nodewright.nvidia.com API group and the NodeWright
kind, and it reads and writes nodewright.nvidia.com/* node annotations. It
requires an operator that serves nodewright.nvidia.com; older Skyhook-only
operators that serve just skyhook.nvidia.com and the Skyhook kind are
unsupported. Upgrade the operator to a NodeWright-capable version before using
this CLI.
Every cluster-backed command runs a preflight check on the served API groups.
If the cluster serves only the legacy skyhook.nvidia.com group and not
nodewright.nvidia.com, the command fails fast with a clear, actionable error
naming both groups and telling you to upgrade to a NodeWright-capable operator,
rather than a confusing NotFound.
The install namespace for new deployments moved from skyhook to nodewright with
the rename. Namespaces cannot be renamed in place, and Helm cannot move a release
between namespaces, so an install that predates the rename legitimately stays in
skyhook indefinitely. The CLI therefore does not simply default to
nodewright.
When --namespace is not passed, the CLI resolves the operator's namespace in
this order and caches the answer for the invocation:
nodewright— if a NodeWright operator Deployment is there, use it.skyhook— if the operator is there instead, use it and print a one-line note to stderr saying the legacy namespace was used and that new installs default tonodewright. Pass--namespace skyhookto silence the note.- A cluster-wide Deployment sweep, for installs in some other namespace. This needs cluster-scoped list permission; if you don't have it, the step is skipped silently rather than failing.
- If nothing is found,
nodewrightis used so the command's own lookup produces the specific error (for example, "no operator deployment found in namespace "nodewright"").
When --namespace is passed, it is used verbatim with no discovery and no
note. Nothing else about the CLI is version-gated on this: the namespace is only
used to locate the operator Deployment and the package pods.
The NodeWright and DeploymentPolicy CRDs are cluster-scoped, so --namespace
never affects which CRs a command sees.
| Situation | --namespace omitted |
--namespace passed |
|---|---|---|
Operator in nodewright |
✅ Discovered | ✅ Used as given |
Operator in skyhook (pre-rename install) |
✅ Discovered, with a deprecation note | ✅ Used as given |
| Operator in some other namespace | ✅ Discovered when the user can list Deployments cluster-wide, otherwise falls back to nodewright and the command reports the miss |
✅ Used as given |
| No operator installed | nodewright; the command reports the miss |
The CLI requires operator version v0.8.0 or later for full functionality of all commands.
| Command | v0.7.x and earlier | v0.8.0+ |
|---|---|---|
version |
✅ Full | ✅ Full |
node status |
✅ Full | ✅ Full |
node list |
✅ Full | ✅ Full |
node reset |
✅ Full | ✅ Full |
node ignore/unignore |
✅ Full | ✅ Full |
package status |
✅ Full | ✅ Full |
package rerun |
✅ Full | ✅ Full |
package logs |
✅ Full | ✅ Full |
reset |
✅ Full | ✅ Full |
reset --package |
✅ Full (v0.7.5+) | ✅ Full |
update-state |
✅ Full (v0.7.5+) — see note | ✅ Full — see note |
deployment-policy reset |
❌ Not supported | ✅ Full |
pause |
❌ Not supported | ✅ Full — stop strength varies, see note |
resume |
❌ Not supported | ✅ Full |
disable |
❌ Not supported | ✅ Full |
enable |
❌ Not supported | ✅ Full |
Note on
pausestop strength: the command works on every v0.8.0+ operator, but how hard it stops is version-dependent, so the column above is about availability only. On operators that run package stages as Jobs, pause suspends the stage that is currently executing. On v0.8.0+ operators that predate that change — and, briefly, for a stage still running as a pre-upgrade pod during the upgrade itself — pause blocks all new stage scheduling but lets an in-flight stage finish. (On v0.7.x and earlier the command does not exist at all; usespec.pause.)disablenever stops in-flight work on any version. See Emergency Stop for the full semantics and for whatresumere-runs.Note on
update-stateandreset --package: These commands edit thenodewright.nvidia.com/nodeState_<nodewright>annotation in-place. The annotation'smap[string]PackageStatusshape has been stable since operator v0.7.5, and the CLI refuses to run against anything older. What has evolved across operator releases is the set of recognised stage and state values — for exampleuninstallanduninstall-interruptwere added in v0.16.0. Picking a stage/state your operator doesn't recognise will leave the package in a state the operator can't progress from. Confirm the stage/state values are valid for your operator before running these commands.
In operator versions v0.7.x and earlier, pausing and disabling a Skyhook was done via spec fields:
spec:
pause: true # Old method - no longer used by operatorStarting with v0.8.0, the operator uses annotations instead:
metadata:
annotations:
nodewright.nvidia.com/pause: "true"
nodewright.nvidia.com/disable: "true"The CLI's pause, resume, disable, and enable commands set these annotations. If you're running an older operator (v0.7.x or earlier), these commands will appear to succeed but the operator won't recognize the annotations - you'll need to edit the Skyhook spec directly using kubectl edit.
# Build from source
cd operator
make build-cli
# Install as kubectl plugin
cp bin/nodewright /usr/local/bin/kubectl-nodewright # or another directory in $PATH with write access
# Verify installation
kubectl nodewright versionkubectl nodewright [global-flags] <command> [subcommand] [flags] [arguments]
-h, --help- Show help for any command--version- Show version information-n, --namespace- Kubernetes namespace the operator is installed in. When omitted, the CLI discovers it (see Namespace resolution); the fallback default isnodewright.-o, --output- Output format: table|json|yaml|wide-v, --verbose- Enable verbose output--dry-run- Preview changes without applying them--kubeconfig- Path to kubeconfig file
Show plugin and operator versions.
# Show both plugin and operator versions
kubectl nodewright version
# Show only plugin version (no cluster query)
kubectl nodewright version --client-only
# With custom timeout
kubectl nodewright version --timeout 10sControl NodeWright processing state.
Note: Requires operator v0.8.0+. See Compatibility for details.
# Pause a Skyhook (stops processing new nodes)
kubectl nodewright pause my-skyhook
kubectl nodewright pause my-skyhook --confirm # Skip confirmation
# Resume a paused Skyhook
kubectl nodewright resume my-skyhookCompletely disable or re-enable a NodeWright.
Note: Requires operator v0.8.0+. See Compatibility for details.
# Disable a Skyhook completely
kubectl nodewright disable my-skyhook
kubectl nodewright disable my-skyhook --confirm
# Re-enable a disabled Skyhook
kubectl nodewright enable my-skyhookReset all package state for a NodeWright, causing re-execution from the beginning.
# Reset all nodes for a Skyhook (also resets batch state by default)
kubectl nodewright reset gpu-init --confirm
# Preview changes without applying (dry-run)
kubectl nodewright reset gpu-init --dry-run
# Reset nodes only, preserve deployment policy batch state
kubectl nodewright reset gpu-init --skip-batch-reset --confirm
# Reset only a single package across all tracked nodes
kubectl nodewright reset gpu-init --package pkg1:1.0 --confirm --skip-batch-reset| Flag | Description |
|---|---|
--confirm, -y |
Skip confirmation prompt |
--skip-batch-reset |
Skip resetting deployment policy batch state |
--package <name>[:<version>] |
Reset only this package's state on each node |
Note: By default,
resetalso resets the deployment policy batch state so the next rollout starts from batch 1, and clears node ordering state (NodeOrderOffsetandNodePriority) soSKYHOOK_NODE_ORDERrestarts from0. Use--skip-batch-resetto preserve the existing batch and ordering state.
When --package is set, reset removes only the named package's entry from
each node's nodeState annotation instead of removing the whole annotation:
- If
<version>is supplied (e.g.pkg1:1.0), only entries whose recorded version matches are removed; nodes with the package at a different version are left untouched and reported in the command output. - If the package was the last entry in the annotation, the entire annotation is removed (matching the behavior of a full reset).
- Batch state is deliberately not reset on this path regardless of
--skip-batch-reset: restarting the full rollout from batch 1 to recover a single package would be disproportionate. Pair withkubectl nodewright deployment-policy resetexplicitly if you also want to restart batches.
Edit the recorded state of a single package on NodeWright-managed nodes.
update-state performs a surgical edit to the per-node
nodewright.nvidia.com/nodeState_<nodewright> annotation — replacing (or, with
--add, inserting) one entry in the map[string]PackageStatus value. It is
an administrator escape hatch for recovering from stuck rollouts and
deliberately does not validate that the requested (stage, state)
combination is one the operator could legally produce, nor does it gate
destructive stages (uninstall, uninstall-interrupt) behind extra
prompts.
kubectl nodewright update-state <skyhook-name> <package> <version> <stage> <state>
⚠️ Pause the NodeWright before runningupdate-state.
update-stateperforms a read-modify-write on the node-state annotation and uses a merge patch with no resource-version check. The patch rewrites the entire annotation value, not just the targeted package's entry — so a concurrent operator (or CLI) edit to any other package's entry in the same annotation will also be clobbered. If the operator reconciles the node between the CLI's read and write, the operator can immediately overwrite the manual edit — at best wasting the operation, at worst racing the operator into an inconsistent state. Always pause the NodeWright (kubectl nodewright pause <name> --confirm) before running this command, and resume only when finished.
update-statealso requires the package + version to still be present inskyhook.Spec.Packages. It cannot edit an orphaned node-state entry left behind after the package was removed from the spec — usereset --package <name>[:<version>]for that.
# Mark pkg1@1.0 as complete on every node that already tracks this Skyhook
kubectl nodewright update-state gpu-init pkg1 1.0 config complete --confirm
# Same, but only on one node
kubectl nodewright update-state gpu-init pkg1 1.0 config complete --node worker-1 --confirm
# Narrow to a label-selected set of nodes
kubectl nodewright update-state gpu-init pkg1 1.0 interrupt in_progress -l role=gpu --confirm
# Preview the changes without writing them
kubectl nodewright update-state gpu-init pkg1 1.0 config erroring --dry-run| Flag | Description |
|---|---|
--node |
Limit the update to specific node(s); repeat for multiple |
--selector, -l |
Limit the update to nodes matching a label selector |
--confirm, -y |
Skip confirmation prompt |
--add |
Create a fresh nodeState entry on nodes that do not yet have one for this package; requires --node or --selector |
The global --dry-run flag is honored: the command prints the set of nodes
it would patch and exits without writing.
By default update-state targets only nodes that already have a nodeState
entry for the named NodeWright. If --node names a node that does not exist
or has no state for the NodeWright, the command warns and skips that node
rather than failing.
--add creates a fresh nodeState entry on nodes that do not yet have one
for the given <package>@<version> — useful for bootstrapping state on a
node the operator has not visited yet, or for re-creating an entry that was
manually deleted.
--add requires either --node or --selector so the scope of the
creation is explicit. Without one of these flags, --add would apply to
every node in the cluster that matches the NodeWright's selector, which is
far too broad for a creation operation — the CLI rejects this combination
with an error.
If a targeted node already has an entry for <package>@<version>, --add
warns and skips that node (use update-state without --add if you intend
to overwrite the existing entry).
There is no dedicated migrate command: the conversion is a mechanical group and
kind swap with no change to the spec body. Both Skyhook and DeploymentPolicy
change API group to nodewright.nvidia.com/v1alpha1; Skyhook additionally
changes its kind to NodeWright, while DeploymentPolicy keeps its kind.
For a source manifest, that is:
sed -e '/^ *apiVersion:/ s|skyhook\.nvidia\.com/|nodewright.nvidia.com/|' \
-e 's|^\( *kind: *\)Skyhook[[:space:]]*$|\1NodeWright|' skyhook.yaml > nodewright.yamlRewrite apiVersion and kind only. A blanket
s|skyhook\.nvidia\.com/|nodewright.nvidia.com/|g is tempting but wrong: it also
rewrites key names inside the spec body that are yours, not the operator's.
nodeSelectors.matchLabels and podNonInterruptLabels refer to labels on your
own nodes and pods; the rename does not touch those labels, so rewriting the
selector points the CR at a key nothing carries and it silently matches nothing.
It also makes the re-apply a real spec change rather than the no-op adoption of
the mirrored object that the migration flow expects.
The one thing to rename by hand, if your manifest sets it, is the operator's own
lifecycle annotations on the CR: skyhook.nvidia.com/pause and
skyhook.nvidia.com/disable become nodewright.nvidia.com/*.
Leave everything else alone. In particular, do not rewrite the runtime-required
taint key here: it is the bare string skyhook.nvidia.com (no trailing
slash), so a blanket group swap would corrupt it, and the taint on a node is not
something a CR manifest edit can change. The default key did move to
nodewright.nvidia.com, but the operator tolerates and removes both keys for the
deprecation window, so an additionalTolerations entry naming the legacy key
stays valid. Migrating it is a change to your provisioning config (autoscaler,
node pool, --register-with-taints), not to your CRs; see
docs/runtime_required.md.
The operator's mirror controller already converts the live objects in the cluster automatically; the edit above is only for the manifests in your source of truth (git / GitOps). See docs/nodewright-migration.md for the full flow.
Manage deployment policy batch state.
Note: Requires operator v0.8.0+.
# Reset batch state for a Skyhook (starts rollout from batch 1)
kubectl nodewright deployment-policy reset gpu-init --confirm
# Preview what would be reset (dry-run)
kubectl nodewright deployment-policy reset gpu-init --dry-run
# Using the short alias
kubectl nodewright dp reset gpu-init --confirmThe deployment-policy reset command resets the batch processing state for all compartments in the specified NodeWright, including:
- Current batch number (reset to 1)
- Consecutive failure count
- Completed and failed node counts
- Stop flag
- Node ordering state (
NodeOrderOffsetandNodePriority) —SKYHOOK_NODE_ORDERrestarts from0
| Flag | Description |
|---|---|
--confirm, -y |
Skip confirmation prompt |
When to use:
- After a rollout completes and you want to start a new rollout fresh
- When batch processing is stuck and needs to be reset
- Before re-running a rollout with the same deployment policy
See Deployment Policy documentation for details on auto-reset configuration.
Manage NodeWright nodes across the cluster.
# List all nodes targeted by a Skyhook
kubectl nodewright node list --nodewright my-nodewright
kubectl nodewright node list --nodewright my-nodewright -o json
# Show all Skyhook activity on specific node(s)
kubectl nodewright node status worker-1
kubectl nodewright node status worker-1 worker-2
kubectl nodewright node status "worker-.*" # Regex pattern
kubectl nodewright node status worker-1 --nodewright my-nodewright # Filter by NodeWright
# Reset all package state on node(s)
kubectl nodewright node reset worker-1 --nodewright my-nodewright --confirm
kubectl nodewright node reset "node-.*" --nodewright my-nodewright --dry-run
# Ignore/unignore nodes from processing
kubectl nodewright node ignore worker-1
kubectl nodewright node ignore "test-node-.*"
kubectl nodewright node unignore worker-1| Command | Flag | Description |
|---|---|---|
list |
--nodewright |
NodeWright name (required) |
status |
--nodewright |
Filter by NodeWright name |
reset |
--nodewright |
NodeWright name (required) |
reset |
--confirm, -y |
Skip confirmation prompt |
Manage NodeWright packages.
# Query package status across nodes
kubectl nodewright package status my-package --nodewright my-nodewright
kubectl nodewright package status my-package --nodewright my-nodewright --node worker-1
kubectl nodewright package status my-package --nodewright my-nodewright -o wide
# Force package re-run on specific nodes
kubectl nodewright package rerun my-package --nodewright my-nodewright --node worker-1
kubectl nodewright package rerun my-package --nodewright my-nodewright --node "worker-.*" --confirm
kubectl nodewright package rerun my-package --nodewright my-nodewright --node worker-1 --stage config
# Get package logs
kubectl nodewright package logs my-package --nodewright my-nodewright
kubectl nodewright package logs my-package --nodewright my-nodewright --node worker-1
kubectl nodewright package logs my-package --nodewright my-nodewright --stage apply
kubectl nodewright package logs my-package --nodewright my-nodewright -f # Follow
kubectl nodewright package logs my-package --nodewright my-nodewright --tail 100| Command | Flag | Description |
|---|---|---|
status |
--nodewright |
NodeWright name (required) |
status |
--node |
Filter by node pattern (repeatable) |
rerun |
--nodewright |
NodeWright name (required) |
rerun |
--node |
Node pattern (required, repeatable) |
rerun |
--stage |
Re-run from stage: apply, config, interrupt, post-interrupt |
rerun |
--confirm, -y |
Skip confirmation prompt |
logs |
--nodewright |
NodeWright name (required) |
logs |
--node |
Filter by node name |
logs |
--stage |
Filter by stage |
logs |
-f, --follow |
Follow log output |
logs |
--tail |
Lines from end (-1 for all) |
# General help
kubectl nodewright --help
# Command group help
kubectl nodewright node --help
kubectl nodewright package --help
kubectl nodewright deployment-policy --help
# Specific command help
kubectl nodewright node reset --help
kubectl nodewright package rerun --help
kubectl nodewright deployment-policy reset --help# 1. Check package status
kubectl nodewright package status my-package --nodewright my-nodewright -o wide
# 2. View logs for the failed package
kubectl nodewright package logs my-package --nodewright my-nodewright --node worker-1
# 3. Fix the issue, then force re-run
kubectl nodewright package rerun my-package --nodewright my-nodewright --node worker-1 --confirm# 1. Ignore node before maintenance
kubectl nodewright node ignore worker-1
# 2. Perform maintenance...
# 3. Unignore and reset to re-run all packages
kubectl nodewright node unignore worker-1
kubectl nodewright node reset worker-1 --nodewright my-nodewright --confirm# List all nodes for a Skyhook
kubectl nodewright node list --nodewright my-nodewright
# Check status of all nodes
kubectl nodewright node status
# Check specific Skyhook across all nodes
kubectl nodewright node status --nodewright my-nodewright -o json# 1. Full reset: nodes + batch state (starts everything fresh)
kubectl nodewright reset my-skyhook --confirm
# 2. Or reset only batch state (keep node state, restart batch progression)
kubectl nodewright deployment-policy reset my-skyhook --confirm
# 3. Or reset only nodes (keep batch progression)
kubectl nodewright reset my-skyhook --skip-batch-reset --confirmWhen a single package on a single node is wedged and a full reset is too disruptive, pause the NodeWright, edit the recorded state directly, then resume:
# 1. Pause so the operator doesn't clobber the manual edit
kubectl nodewright pause my-skyhook --confirm
# 2. Mark the wedged package as complete (or set whatever stage/state
# you need to unblock the rollout)
kubectl nodewright update-state my-skyhook my-package 1.0 config complete \
--node worker-1 --confirm
# 3. Resume processing
kubectl nodewright resume my-skyhook --confirmFor a single-package reset across all nodes (without disturbing the
deployment policy batch state), prefer reset --package:
kubectl nodewright reset my-skyhook --package my-package:1.0 --confirmNote: Requires operator v0.8.0+. For older operators, use
kubectl edit skyhook my-skyhookand setspec.pause: true.
pause and disable stop different things. pause is the one that can halt work already running: on operators that run package stages as Jobs it suspends the executing stage, and on older ones it blocks new scheduling only — see the version note below. disable takes the NodeWright out of processing so no new work is scheduled, but it does not stop a stage already under way; that stage runs to completion.
They compose safely in either order: disabling a paused NodeWright leaves its suspended stages suspended rather than resuming them. Those stages resume only once both annotations are cleared — re-enabling on its own does nothing while the NodeWright is still paused. If you want everything stopped now, pause is the command.
# Pause all processing
kubectl nodewright pause my-skyhook --confirm
# Or prevent new work being scheduled
kubectl nodewright disable my-skyhook --confirmHow hard pause stops depends on the operator version. On operators that run package stages as Jobs, pause suspends the stage that is currently executing — its pod is signaled to stop and nothing new starts until you
resume, at which point the stage re-runs from the start of its current phase (the agent skips already-completed steps). On earlier operators — and, briefly, for a stage still running as a pre-upgrade pod during an operator upgrade — a stage already in flight finishes its current run before pause takes hold; pause still blocks all new stage scheduling. If you need a running stage to stop immediately, confirm the operator executes packages as Jobs.
All status commands support multiple output formats:
# Table (default) - human-readable
kubectl nodewright node list --nodewright my-nodewright
# Wide - table with additional columns
kubectl nodewright node list --nodewright my-nodewright -o wide
# JSON - machine-readable
kubectl nodewright node list --nodewright my-nodewright -o json
# YAML - machine-readable
kubectl nodewright node list --nodewright my-nodewright -o yamloperator/cmd/cli/app/ # CLI commands
├── cli.go # Root command (NewSkyhookCommand)
├── version.go # Version command
├── reset.go # Reset command (nodes + batch state)
├── lifecycle.go # Pause, resume, disable, enable commands
├── deploymentpolicy/ # Deployment policy subcommands
│ ├── deploymentpolicy.go # Parent command
│ └── deploymentpolicy_reset.go # Batch state reset
├── node/ # Node subcommands
│ ├── node.go # Parent command
│ ├── node_list.go
│ ├── node_status.go
│ ├── node_reset.go
│ └── node_ignore.go # Ignore and unignore commands
└── package/ # Package subcommands
├── package.go # Parent command
├── package_status.go
├── package_rerun.go
└── package_logs.go
operator/internal/cli/ # Shared CLI utilities
├── client/ # Kubernetes client wrapper
├── context/ # CLI context and global flags
└── utils/ # Shared utilities
main()
└── cli.Execute()
└── NewSkyhookCommand(ctx)
├── NewVersionCmd(ctx)
├── NewResetCmd(ctx)
├── NewPauseCmd(ctx)
├── NewResumeCmd(ctx)
├── NewDisableCmd(ctx)
├── NewEnableCmd(ctx)
├── deploymentpolicy.NewDeploymentPolicyCmd(ctx)
│ └── NewResetCmd(ctx)
├── node.NewNodeCmd(ctx)
│ ├── NewListCmd(ctx)
│ ├── NewStatusCmd(ctx)
│ ├── NewResetCmd(ctx)
│ ├── NewIgnoreCmd(ctx)
│ └── NewUnignoreCmd(ctx)
└── pkg.NewPackageCmd(ctx)
├── NewStatusCmd(ctx)
├── NewRerunCmd(ctx)
└── NewLogsCmd(ctx)
# Run CLI tests
make test-cli
# Run all tests
make test