Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 46 additions & 11 deletions agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,39 @@ A basic example of using a container overlay

The Go rewrite under `agent/go` shares execution policy between steps and
interrupts through `execution.Config`. A `Config` composes the host root mount,
the child-visible step and package directories, and the stdout and stderr
writers that receive raw command output. Operations report `execution.Status`:
the package directories inside that host, and the stdout and stderr writers
that receive raw command output. Non-host steps resolve those directories
through the mounted host root before execution. Operations report
`execution.Status`:
`execution.StatusSuccess` means the operation satisfied its execution policy,
while `execution.StatusFailed` means it did not.

Each Go interrupt owns its command construction and execution. The `Interrupt`
contract exposes `Type` for the wire identity, `Run` for execution using an
`execution.Config`, and `Serialize` for the operator-facing representation.
Retry state and completion flags remain orchestration concerns outside the
interrupt implementations.
The orchestration layer writes one completion marker per interrupt type and
resource ID after successful execution. Node restarts use a pending marker
containing the host boot ID: a changed boot ID promotes the marker to complete,
while an unchanged boot ID retries the restart. This keeps reboot completion
independent of the signal used to terminate the agent or its child process.

The Go entrypoint accepts the current operator forms:

```text
agent MODE ROOT_MOUNT COPY_DIR
agent interrupt ROOT_MOUNT COPY_DIR INTERRUPT_DATA
```

It also accepts the legacy forms, which default `ROOT_MOUNT` to `/root`:

```text
agent MODE COPY_DIR
agent interrupt COPY_DIR INTERRUPT_DATA
```

SIGTERM cancels the active step or interrupt and prevents later steps from
starting. A failed operation or runtime error exits with status 1; malformed
arguments exit with status 2.

### Container Image Build

Expand All @@ -40,14 +63,26 @@ interrupt implementations.

## Environment variables

There are a number of environment variables that can be used to control how the controller works
There are a number of environment variables that can be used to control how the agent works.

1. `COPY_RESOLV` if set to `"false"` it will NOT copy the container's `/etc/resolv.conf` to the host.
1. `OVERLAY_ALWAYS_RUN_STEP` if set to `"true"` it will ignore any step flags and always run every step. A warning will be printed to stdout if it sees a flag file.
1. `SKYHOOK_AGENT_BUFFER_LIMIT` defaults to 8KB. This is how much of the log of each step it will read before syncing the data to stdout/stderr and the log file. It is recommended to keep this somewhat low to avoid excessive delay between a step emitting some information and seeing it in the docker logs or in the log file.
1. `OVERLAY_ALWAYS_RUN_STEP` if set to `"true"` it will ignore any step flags and always run every step. A warning is logged if it sees a flag file.
1. `SKYHOOK_AGENT_WRITE_LOGS` defaults to `"true"`. Step and interrupt output is streamed directly to stdout/stderr and also written under `SKYHOOK_LOG_DIR`. Set it to `"false"` to stream without retaining host log files.

`SKYHOOK_AGENT_BUFFER_LIMIT` applies only to the Python agent. The Go agent
streams command output directly and does not buffer it.

The following environment variables are required and are expected to be set by either the build system or skyhook-operator. It is not recommended that they be changed manually.

1. `OVERLAY_FRAMEWORK_VERSION` is the version of the current overlay. It is expected that this gets set by the docker build system. It is required to be able to manage the history file. It must be in the format of `{package name}-{version}`.
1. `SKYHOOK_RESOURCE_ID` is used to determine if an interrupt should be rerun. Interrupts are only run once per `SKYHOOK_RESOURCE_ID`. Skyhook operator should make this unique per configuration of the package.

The following environment variables are optional and use the documented defaults when unset:

1. `SKYHOOK_DATA_DIR` is the package data source used by legacy invocations when the operator has not already populated `COPY_DIR`. It defaults to `/skyhook-package`.
1. `SKYHOOK_ROOT_DIR` is the host state root for flags, interrupt markers, and history. It defaults to `/etc/skyhook`.
1. `SKYHOOK_LOG_DIR` is the host log root. It defaults to `/var/log/skyhook`.

The following are enviroment variables expected to be set by either the build system or skyhook-operator. It is not recommended they be changed manually.
The following environment variable is optional:

1. `OVERLAY_FRAMEWORK_VERSION` this the version of the current overlay. It is expected that this gets set by the docker build system. It is required to be able to manage the history file. It must be in the format of `{package name}-{version}`
1. `SKYHOOK_RESOURCE_ID` this is used to determine if an interrupt should be rerun. Interrupts are only run once per `SKYHOOK_RESOURCE_ID`. Skyhook operator should make this unique per conifguration of the package.
1. `SKYHOOK_NODE_ORDER` zero-indexed monotonic position of this node in the rollout. The first batch's nodes get `0, 1, 2, ...` and subsequent batches continue from where the previous batch left off. Useful for kubeadm upgrade workflows where the first node (`SKYHOOK_NODE_ORDER=0`) runs a different command than subsequent nodes. See [Node Order Within a Rollout](../docs/ordering_of_skyhooks.md#node-order-within-a-rollout) for details.
1. `SKYHOOK_NODE_ORDER` is a zero-indexed monotonic position of this node in the rollout. The first batch's nodes get `0, 1, 2, ...` and subsequent batches continue from where the previous batch left off. Useful for kubeadm upgrade workflows where the first node (`SKYHOOK_NODE_ORDER=0`) runs a different command than subsequent nodes. See [Node Order Within a Rollout](../docs/ordering_of_skyhooks.md#node-order-within-a-rollout) for details.
19 changes: 11 additions & 8 deletions agent/go/cmd/agent/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,17 +19,20 @@
package main

import (
"fmt"
"log/slog"
"context"
"os"
"os/signal"
"syscall"

"github.com/NVIDIA/nodewright/agent/internal/agent"
)

func main() {
// Establish the agent's structured-logging seam. Packages (e.g.
// config.Loader.Load) log through *slog.Logger and fall back to
// slog.Default() when passed nil, so wiring it here once keeps that
// default sane for the whole process.
slog.SetDefault(slog.New(slog.NewTextHandler(os.Stderr, nil)))
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer stop()

nodewrightAgent := agent.New()
exitCode := nodewrightAgent.Run(ctx, os.Args[1:], os.Stdout, os.Stderr)

fmt.Println("Hello, World!")
os.Exit(int(exitCode))
}
Loading
Loading