feat(fleetcli): protobuf-driven Fleet command-line client#467
feat(fleetcli): protobuf-driven Fleet command-line client#467krisztiankurucz wants to merge 30 commits into
Conversation
github-actions
Bot
added
documentation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Introduces a protobuf-driven generator for fleetcli, adds handwritten runtime support (auth/session, pairing streaming, firmware HTTP lifecycle), and expands automated validation via unit + e2e tests.
Changes:
- Add
server/tools/generate-fleet-cligenerator (templates, overrides, report) and wire it intojust gen. - Implement/extend
server/cmd/fleetcliruntime (auth resolution, generated command runtime, pairing + firmware command groups). - Add extensive unit tests and e2e coverage for the CLI workflows.
Reviewed changes
Copilot reviewed 24 out of 36 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| server/tools/generate-fleet-cli/templates/fleet-group.gotmpl | Template for generated per-group command files. |
| server/tools/generate-fleet-cli/templates/fleet-commands.gotmpl | Template for generated command registry file. |
| server/tools/generate-fleet-cli/overrides.json | Declarative mapping of services/methods to CLI groups and naming/auth overrides. |
| server/tools/generate-fleet-cli/main.go | Core generator: loads descriptor set + overrides, analyzes requests, renders Go sources and a coverage report. |
| server/tools/generate-fleet-cli/main_test.go | Unit tests for goPackageInfo behavior. |
| server/go.mod | Adds direct deps used by fleetcli output/formatting and cli framework. |
| server/go.sum | Records checksums for new dependencies. |
| server/e2e/plugin_integration_test.go | Updates integration test auth flow (cookie session -> API key) and modernizes selector/telemetry calls. |
| server/e2e/curtailment_e2e_test.go | Switches proto-sim discovery host to container-reachable hostname. |
| server/e2e/fleetcli_e2e_test.go | New e2e test for core CLI workflow (onboarding -> discover -> pair -> miners -> performance). |
| server/e2e/fleetcli_firmware_e2e_test.go | New e2e test covering firmware lifecycle via CLI. |
| server/e2e/fleetcli_subcommands_e2e_test.go | New broad e2e test exercising many CLI subcommands and asserting leaf-command coverage. |
| server/docs/fleet-cli-generation.md | Documents generator architecture, inputs, statuses, and workflow. |
| server/cmd/fleetcli/main.go | Root CLI + manual command groups (auth/apikey/performance/firmware) and shared helpers. |
| server/cmd/fleetcli/client.go | CLI HTTP/Connect client, auth/session handling, URL normalization, secure-cookie loopback support. |
| server/cmd/fleetcli/generated_runtime.go | Shared runtime invoked by generated command expressions (auth dispatch + selector helpers). |
| server/cmd/fleetcli/pairing.go | Handwritten pairing group (streaming discover + selector-based pair). |
| server/cmd/fleetcli/firmware_client.go | Firmware REST client for config/check/upload/list/delete endpoints. |
| server/cmd/fleetcli/firmware.go | Handwritten firmware command group and upload orchestration + progress reporting. |
| server/cmd/fleetcli/manual.go | Aggregates handwritten commands for registry composition. |
| server/cmd/fleetcli/main_test.go | Unit tests for auth flag/env resolution + enum normalization + regression cases. |
| server/cmd/fleetcli/client_test.go | Unit tests for base URL normalization and loopback secure-cookie semantics. |
| server/cmd/fleetcli/pairing_test.go | Unit tests for discover/pair request builders and selector/credential behaviors. |
| server/cmd/fleetcli/firmware_test.go | Unit tests for firmware config/check/upload modes, chunking, validation, and error shaping. |
| justfile | Adds gen-fleet-cli / build-fleet-cli and runs generator as part of gen. |
| bin/scripts/fleetcli | Convenience script that builds and execs the local CLI binary. |
| bin/hermit.hcl | Adds scripts directory to PATH for hermit-managed environments. |
Files not reviewed (9)
- server/cmd/fleetcli/cmd_commands.go: Generated file
- server/cmd/fleetcli/cmd_groups.go: Generated file
- server/cmd/fleetcli/cmd_minercommand.go: Generated file
- server/cmd/fleetcli/cmd_miners.go: Generated file
- server/cmd/fleetcli/cmd_networkinfo.go: Generated file
- server/cmd/fleetcli/cmd_onboarding.go: Generated file
- server/cmd/fleetcli/cmd_pools.go: Generated file
- server/cmd/fleetcli/cmd_racks.go: Generated file
- server/cmd/fleetcli/cmd_schedule.go: Generated file
krisztiankurucz
force-pushed
the
kkurucz/port-fleet-client-api
branch
from
c202bcd to
d2f92ad
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c202bcd920
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6f78fad713
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e2a7aed9bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05605e994a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
ankitgoswami
left a comment
ankitgoswami
left a comment
There was a problem hiding this comment.
I’m very supportive of having a CLI. I think it would be useful for operators, scripting, debugging, and workflows where the UI is too heavy. My hesitation is mostly about timing and where the CLI should attach.
RFC 0001 moves plugin orchestration, miner I/O, discovery, pairing, and command execution toward fleet nodes. Given that, I think the long-term CLI probably wants to be designed around the fleet-node boundary rather than around today’s transitional server/combined-mode API shape.
The main upside of making the CLI run through fleet nodes is that it would match the architecture we’re already moving toward: commands execute near the miners, LAN-only devices stay behind the customer network, local/degraded-mode workflows become possible, and the CLI semantics line up with fleet-node ownership instead of cloud reachability. It also gives us a cleaner story for multi-site and air-gapped setups.
The trade-off is that it likely means waiting for more of the fleet-node surface to settle, and the first version may be less broad. There’s also more complexity around node selection, enrollment state, offline behavior, and explaining which operations are server-wide versus scoped to a specific node. But I think that complexity is inherent in the product direction anyway, so baking it into the CLI from the start seems better than shipping a large CLI now and reshaping it soon after.
This fleetcli tool is a dependency for some additional testrack / rig reservation tooling I’m working on, so I still need a CLI surface for higher-level server / control-plane actions. That said, I agree with the concern about not baking in today’s transitional server-centric model for miner-local operations. For this first version, I’m happy to narrow the scope to control-plane workflows only, and remove or defer miner I/O, discovery, pairing, and command execution while the fleet-node command surface solidifies. Concretely, I’d keep things like auth/API key management, onboarding, groups/racks metadata, saved pool/config metadata, and read-only server state where appropriate. I’d defer pairing, minercommand, LAN/network discovery, and any command paths that imply direct miner execution or node-involved operations. Longer term, once the fleet-node API is settled, we can bring those operational commands back with explicit node/site scoping instead of treating the server API as the permanent CLI boundary. Does that make sense? |
|
yea, that makes sense to me. another thing to note here is that fleetnodes already have a CLI and will essentially be the way to communicate with a cloud hosted version of fleet. I'll review this once you have updated it. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 27cba3910b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b101aae16a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
krisztiankurucz
force-pushed
the
kkurucz/port-fleet-client-api
branch
from
7a8bf02 to
8e77b89
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e77b898d9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 778d236665
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d5acee6760
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 15138aac62
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9965e10fbc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e902a3401e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
krisztiankurucz
force-pushed
the
kkurucz/port-fleet-client-api
branch
from
d6b0dc7 to
5faed94
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 083419a4e7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
…ing unchanged files Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
…atch Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
…nputs through Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
…e build output Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
Signed-off-by: Krisztian Kurucz <kkurucz@squareup.com>
krisztiankurucz
force-pushed
the
kkurucz/port-fleet-client-api
branch
from
d6ea09a to
4cdd62b
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4cdd62b329
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99a9c40548
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| &cli.StringFlag{Name: "json", Usage: "Path to a request JSON file, or - for stdin"}, | ||
| &cli.Int64Flag{Name: "collection-id", Usage: "collection id"}, | ||
| &cli.StringFlag{Name: "label", Usage: "label"}, | ||
| }, generatedCommonSelectorFlags()...), |
There was a problem hiding this comment.
When racks save is combined with a JSON rack body and --all-devices, this default selector helper overwrites the JSON device_selector with DeviceSelector_AllDevices. SaveRack resolves that selector to every paired device before replacing rack membership, so a user can move the whole fleet into one rack (and cascade placement) via a flag that was already removed from racks add-devices; use the device_list selector mode or reject --all-devices for rack saves in the generator override.
Useful? React with 👍 / 👎.
Summary
Adds
fleetcli, a new command-line client for Fleet control-plane workflows (server/cmd/fleetcli). Most of its command surface is generated from the repo's protobuf definitions by a new code generator (server/tools/generate-fleet-cli); a small handwritten runtime supplies the HTTP/JSON transport, auth, output formatting, and the commands that don't map cleanly onto a single RPC. The client speaks the existing Connect-style HTTP+protojson surface through the nginxapi-proxy, so it needs no separate gRPC dialing or credentials.This is a deliberately scoped V0: it covers auth, API keys, onboarding, groups, racks, pools, miner listing, performance reads, and the firmware file lifecycle. Broad miner I/O, discovery/pairing, schedules, and network info are intentionally deferred while the fleet-node command model settles. Firmware deployment is the one narrow RPC exception, gated to explicitly named devices, groups, or racks (never all devices).
How it works
Two layers compile into one binary:
cmd_<group>.go,cmd_commands.go).just gen-fleet-clirunsbuf build --as-file-descriptor-setto produce a descriptor set, then runs the generator. The generator reads the descriptor set plusoverrides.json, classifies each RPC into an output mode, renders Go command source through two templates, and writes a coverage report.main.go,client.go,generated_runtime.go,firmware.go,firmware_client.go,manual.go). This owns global flags, the HTTP/JSON transport and auth dispatch, request-building helpers shared by generated commands, colorized output, and the commands that don't fit the generated RPC shape (auth,apikey,performance,firmware).overrides.jsondeclares the public surface that descriptors can't encode: which services/methods to expose, group and subcommand names, per-command auth mode, and fixed/ignored request fields. A notable use:groupsandracksare two separate CLI groups both backed bycollection.v1.DeviceCollectionService, disambiguated only by a fixedtypefield — the generator's "split one service across multiple CLI groups" capability.The transport (
client.go) marshals each request with protojson andPOSTs it to{base}/{package}.{Service}/{Method}. A bare--serverhost defaults to the/api-proxyroute; a trailing slash targets the RPC root directly for localfleet-api. Three auth modes are dispatched per command:anonymous(no credentials),bearer(API key if set, else fall back to a username/password cookie session), andsession(cookie session only). A cookie jar wrapper treats loopbackhttp://origins as secure so theSecure-flagged session cookie works against a localfleet-api. Auth inputs are always read from the root command, so a subcommand-local flag that happens to be namedusername/password(e.g. pool credentials) can never leak into Fleet auth.firmwareis handwritten because the file lifecycle is plain JSON/multipart/ chunked HTTP under/api/v1/firmware/(session-cookie auth), not protobuf RPC:config,check,upload,list,delete,delete-all.uploadSHA-256-hashes the file and reuses an existing server copy on a checksum hit (unless--force), then picks a direct or chunked upload by size.firmware deployis the single RPC exception: it requires--firmware-file-idplus at least one bounded selector (--device,--group/--group-id,--rack/--rack-id). Group and rack selectors are expanded client-side (paginated) to their member device identifiers via the shared selector resolver, so theminercommand.v1.FirmwareUpdateRequestalways carries an explicitinclude_deviceslist.--all-devicesis intentionally not a deploy flag, and the command calls onlyMinerCommandService/FirmwareUpdate.Generation pipeline (build time)
flowchart LR proto["proto/ modules"] -->|buf build| desc["fleet-descriptor-set.bin"] overrides["overrides.json"] --> gen["generate-fleet-cli"] desc --> gen gen --> cmds["cmd_GROUP.go + cmd_commands.go"] gen --> report["fleet-cli-report.json (per-RPC status)"] cmds --> bin["fleetcli binary"] runtime["handwritten runtime (main, client, generated_runtime, firmware)"] --> binRequest and auth flow (run time)
flowchart TD user["fleetcli GROUP COMMAND"] --> reg{"generated or handwritten?"} reg -->|generated| grt["generated_runtime: build request from flags or --json"] reg -->|handwritten| man["auth / apikey / performance / firmware"] grt --> auth{"auth mode"} man --> auth auth -->|anonymous| anon["no credentials"] auth -->|bearer| bear["API key, else username/password cookie session"] auth -->|session| sess["cookie session only"] anon --> http["POST protojson to api-proxy RPC path"] bear --> http sess --> http http --> fleet["fleet-api"] fleet --> out["protojson / JSON response, colorized to stdout"]Firmware lifecycle
flowchart TD up["firmware upload PATH"] --> cfg["GET firmware config (limits, chunk size)"] cfg --> val["validate extension + size, SHA-256"] val --> chk{"checksum already on server?"} chk -->|yes, no --force| reuse["reuse existing file id"] chk -->|no| size{"size within chunk size?"} size -->|yes| direct["multipart direct upload"] size -->|no| chunk["chunked initiate / chunk / complete"] dep["firmware deploy --firmware-file-id (--device / --group / --rack ...)"] --> resolve["expand groups/racks to member device ids, client-side"] resolve --> fwreq["FirmwareUpdateRequest, explicit include_devices"] fwreq --> fwrpc["MinerCommandService/FirmwareUpdate"]V0 command surface
groups(full CRUD + members/stats/device),racks(CRUD + slots/zones/types/members/stats),miners list,onboarding create-admin, fullpoolsgroupauth login,apikey create/list/revoke(session-only),performance get(telemetry summary),firmware config/check/upload/list/delete/delete-all/deployminercommand/networkinfo/schedulesurfacesAreas of the code involved
server/tools/generate-fleet-cli/main.go,templates/*.gotmplserver/tools/generate-fleet-cli/overrides.jsonserver/cmd/fleetcli/cmd_*.gogroups,miners,onboarding,pools,racks).server/cmd/fleetcli/generated_runtime.goserver/cmd/fleetcli/client.go--serverURL normalization,APIErrorwith raw body.server/cmd/fleetcli/main.goauth/apikey/performance, protojson/JSON output helpers, auth inputs read from root command.server/cmd/fleetcli/firmware.go,firmware_client.gofirmware deployRPC wrapper (devices, or groups/racks expanded to devices via the shared bounded selector;--all-devicesdisallowed).server/cmd/fleetcli/manual.goserver/cmd/fleetcli/*_test.goserver/e2e/fleetcli*_test.go,plugin_integration_test.gojustfile,bin/scripts/fleetcli,bin/hermit.hclgen-fleet-cli/build-fleet-clitargets (output under.cache/to avoid watch restarts), a build-then-exec wrapper on the Hermit PATH.server/go.mod,server/go.sumurfave/cli/v3,fatih/color,hokaccha/go-prettyjson.server/docs/fleet-cli-generation.mdKey technical decisions & trade-offs
api-proxyroute and protojson encoding, so there's no separate gRPC dial, credential plumbing, or proto-wire handling. The cost is that the CLI speaks the Connect-style POST surface rather than native gRPC streaming, which is one reason streaming RPCs are deferred.just gen-fleet-cli, while genuinely non-RPC or UX-shaped commands (performance, the firmware file lifecycle) stay handwritten instead of being forced through the generator.generated,generated_json_fallback,generated_json_only, or adeferred_*reason) and recorded infleet-cli-report.json, making the generator boundary auditable rather than silent.--all-devicesis intentionally not a deploy flag, so a script cannot trigger a fleet-wide firmware update.username/password) from leaking into Fleet authentication.server/watch never restartsfleet-apiover a no-op regen.Testing & validation
Added unit tests (
cmd/fleetcli: transport, output, firmware lifecycle, deploy request/validation, deploy group→devices resolution, and--all-devicesrejection;tools/generate-fleet-cli: generator output) and E2E tests (fleetcli_e2e,fleetcli_subcommands_e2e,fleetcli_firmware_e2e, plus sharedplugin_integrationharness changes).Validated:
just gen-fleet-cli(regenerate; confirm no unexpected diff)cd server && go test -count=1 ./cmd/fleetclicd server && go test -count=1 ./tools/generate-fleet-clicd server && go test -count=1 -tags e2e ./e2e -run TestFleetCLILeafCommandCoveragegit diff --checkNot re-run after the V0 narrowing: the live docker-compose E2E workflows. The
e2epackage was compiled under thee2ebuild tag and the leaf-command coverage test was executed.