[History Server] Poll Serve, placement group, and Ray Data endpoints by default by Future-Outlier · Pull Request #5093 · ray-project/kuberay

Future-Outlier · 2026-08-05T22:36:26Z

Why are these changes needed?

The History Server rebuilds the UI from Ray events, but Serve applications, placement groups, and Ray Data datasets are not in the event stream yet. The head collector now polls these endpoints by default so replayed clusters show them:

/api/serve/applications/
/api/v0/placement_groups?detail=1&limit=10000
/api/data/datasets/{job_id} (per job discovered via /api/jobs/)

You can set the interval with RAY_COLLECTOR_POLL_INTERVAL if you want (default 30s). RAY_COLLECTOR_ADDITIONAL_ENDPOINTS still adds more endpoints on top. Ray Train can follow once its frontend schema is settled.

Screenshots

Dead clusters replayed through the History Server, each endpoint with and without data.

Serve applications:

Serve applications, cluster that never ran Serve:

Ray Data datasets from an illustrative larger manual workload with Prometheus (separate from the small E2E fixture), with a row expanded into its operators:

Placement groups; this job used no Ray Data, so no Ray Data Overview section:

Placement groups, job that created none:

Related issue number

Labels

If this PR has user-facing changes that require documentation updates at release time, I have added the doc-updates-required label.
If this PR contains breaking changes, I have added the breaking-change label.

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests

Manual test instructions

Unit: cd historyserver && make test.
E2E on kind + MinIO: cd historyserver && go test -count=1 ./test/e2e/ -run 'TestCollector$' — 8/8 passing.
The Ray Data memory and CPU columns come from Prometheus queries in the dashboard, so they read 0 without a Prometheus server (on live clusters too); the screenshot was taken with one installed.

…by default The head collector now polls these endpoints itself every RAY_COLLECTOR_POLL_INTERVAL (default 30s); RAY_COLLECTOR_ADDITIONAL_ENDPOINTS still adds more on top.

RayServiceReconciler.Recorder is events.EventRecorder, so record.NewFakeRecorder does not compile. Every other test in this file already uses events.

A periodic poll now stores an empty Serve response since that is the live truth, while the final shutdown poll still cannot erase a converged snapshot. Periodic polling stops on the shutdown signal instead of ShutdownChan so no tick can overwrite the final pass.

Signed-off-by: Future-Outlier <eric901201@gmail.com>

- Cap the shutdown wait on the periodic poller so a stuck storage write cannot consume the termination grace period, and skip periodic writes once shutdown starts so the final poll owns the store. - Drop the placement-group replay from the collector suite: testDeadClusterPlacementGroups already covers the same round trip. - Kill and reap the e2e kubectl port-forward on cleanup: a leaked forward keeps the port and the next test silently talks to a deleted namespace. Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier · 2026-08-06T01:30:19Z

cc @win5923 @machichima to take a look, tks

machichima

Only went through part of the PR, will finish the review tomorrow!

…ndpoints

- Match the sibling env blocks when parsing RAY_COLLECTOR_POLL_INTERVAL. - Merge the built-in and configured endpoints with slices.Concat. - Describe what PollAdditionalEndpointsPeriodically and pollAllEndpoints do, and keep the shutdown-ordering rationale at the call site. Signed-off-by: Future-Outlier <eric901201@gmail.com>

Signed-off-by: Future-Outlier <eric901201@gmail.com>

machichima

Overall LGTM, just few comments

Signed-off-by: Future-Outlier <eric901201@gmail.com>

machichima · 2026-08-09T03:40:10Z

Just to confirm, I guess RAY_COLLECTOR_POLL_INTERVAL can only be 30s? I saw you change it back to 30s in: f4f0c84

machichima

Overall LGTM! Tested with ray-data.yaml and rayservice.yaml on my side. Just two comments

Future-Outlier · 2026-08-09T14:42:08Z

Just to confirm, I guess RAY_COLLECTOR_POLL_INTERVAL can only be 30s? I saw you change it back to 30s in: f4f0c84

It can be less than 30s, I just want to keep the convention.
update: I decided to remove this and increase TTL seconds

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier · 2026-08-09T15:39:43Z

CI note: the failed Build historyserver attempts (previous, latest) both hit the existing flaky pkg/collector/eventcollector/TestShutdown_DrainsRetriedTasks. This PR does not modify that package.

In both logs, shutdown closes stopProducers before retryProcess registers its final attempt (Giving up ... during shutdown; keys: []). The previous failure passed on a same-HEAD rerun, and the latest race also reproduces locally with -count=500. No PR code change is needed.

Signed-off-by: Future-Outlier <eric901201@gmail.com>

win5923

Thank you!

Signed-off-by: Future-Outlier <eric901201@gmail.com>

andrewsykim · 2026-08-10T15:23:16Z

 	sessionName := path.Base(activeSessionDir)

+	// The head collector shares a pod with the dashboard; override for non-default ports.
+	dashboardAddress := "http://localhost:8265"


can we put this into a constant and/or default it via a flag as well?

Addressed in 0e42467. http://localhost:8265 is now defaultDashboardAddress, and --dashboard-address uses that default. When both are set, RAY_DASHBOARD_ADDRESS intentionally wins, matching the neighboring collector env overrides; the help text and four precedence cases make this explicit.

andrewsykim · 2026-08-10T15:26:19Z

Can we improve this function name and the comments a bit?

Addressed in 0e42467. I renamed the helper to missingSnapshotFallback and clarified its contract: when a dead session has no stored snapshot, known frontend paths receive a schema-valid empty payload, while unknown paths preserve the 404. Focused tests cover Serve, Ray Data, an unrelated path, a missing job ID, and a path with an extra segment such as /api/data/datasets/01000000/unknown.

andrewsykim · 2026-08-10T15:26:51Z

 		return data
 	default:
+		// Nothing is stored for a job that never used Ray Data; match the live dashboard.
+		if strings.HasPrefix(trimmed, "/api/data/datasets/") {


What's the reason for not including train endpoints?

train endpoint is not in the ray's frontend now

Built-in polling is scoped to endpoints consumed by the replayed Ray Dashboard UI, so we avoid storing snapshots the UI will not read. Train can still be configured through RAY_COLLECTOR_ADDITIONAL_ENDPOINTS; we can make it built-in if the supported frontend starts consuming it.

Kept Train opt-in in 0e42467. The Ray 2.56 replayed frontend does not request this endpoint, while users can still add /api/train/v2/runs/v1 through RAY_COLLECTOR_ADDITIONAL_ENDPOINTS. I also aligned the stale RayJob example to show Train as optional rather than re-adding the now-built-in placement-group endpoint.

andrewsykim · 2026-08-10T15:27:30Z

 	// Post-process placement_groups response to add missing fields that the frontend requires.
 	// The collector may not store "bundles" and "stats", but the frontend crashes without them.
 	trimmedPath := strings.TrimRight(req.Request.URL.Path, "/")
 	if trimmedPath == "/api/v0/placement_groups" {


how come we don't need to do something similar for other endpoints (serve, data, etc)?

Updated in 0e42467. The replay shim is now ensurePlacementGroupBundles and only backfills a missing bundles field: Ray 2.56 calls bundles.map() without a guard, while optional stats is guarded and remains untouched. Stored Serve and Ray Data payloads otherwise replay as-is, with schema fallbacks only when no snapshot exists. Tests verify the bundles-only change and byte-for-byte passthrough for an already complete payload.

andrewsykim · 2026-08-10T15:28:15Z

 		})
 		return data
 	default:
+		// Nothing is stored for a job that never used Ray Data; match the live dashboard.


this comment is a bit confusing, how do we know at this point that a job never used Ray Data?

Updated in 0e42467. The comment now states only that no snapshot was stored: Ray can evict finished dataset stats and then return the live empty {"datasets":[]} response; collection can also fail before a non-empty response is persisted. Therefore absence does not prove the job never used Ray Data. The fallback applies only to an exact /api/data/datasets/{jobID} path; tests keep /api/data/datasets/ and paths with an extra segment such as /api/data/datasets/01000000/unknown on the normal 404 path.

andrewsykim · 2026-08-10T15:29:51Z

 	// triggers "Serve not started" warning, matching live cluster behavior.
 	// Ref: https://github.com/ray-project/ray/blob/27d3d81d47/python/ray/dashboard/client/src/pages/serve/hook/useServeApplications.ts#L30
 	// Ref: https://github.com/ray-project/ray/blob/b775a604ce/python/ray/dashboard/client/src/pages/serve/ServeDeploymentsListPage.tsx#L68
 	case "/api/serve/applications":


I think in general we can improve the discovery of "default additional collector endpoints" somehow. Maybe a variable or constant that stores the defaults?

yes we can do that, but in the future we will remove that, since ray will export those endpoint's event

Addressed in 0e42467. The fixed built-in set is now named defaultPolledEndpoints and contains Serve applications and placement groups. Ray Data is also built in, but its per-job paths are discovered through jobsEndpoint and handled by pollDataDatasets, so it intentionally is not represented as a fixed URI in that slice. Existing tests distinguish the fixed defaults from configured endpoints and verify deduplication.

…-endpoints # Conflicts: # historyserver/pkg/collector/logcollector/runtime/logcollector/poll.go # historyserver/test/e2e/collector_test.go

Signed-off-by: Future-Outlier <eric901201@gmail.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 0e42467. Configure here.}

Signed-off-by: Future-Outlier <eric901201@gmail.com>

andrewsykim · 2026-08-11T02:42:18Z

 	flag.DurationVar(&pushInterval, "push-interval", time.Minute, "")
 	flag.StringVar(&ownerKind, "owner-kind", "", "")
 	flag.StringVar(&ownerName, "owner-name", "", "")
+	flag.StringVar(&dashboardAddress, "dashboard-address", defaultDashboardAddress, "Base URL of the Ray Dashboard; overridden by RAY_DASHBOARD_ADDRESS when set")


nit: ray-dashboard-address

andrewsykim · 2026-08-11T03:15:32Z

 	if val := os.Getenv("STORAGE_BACKEND_CONFIG_PATH"); val != "" {
 		storageBackendConfigPath = val
 	}
+	dashboardAddress = resolveDashboardAddress(dashboardAddress)


this function seems overkill just do this?

if value := os.Getenv("RAY_DASHBOARD_ADDRESS"); value != "" { dashboardAddress = val }

andrewsykim · 2026-08-11T03:16:22Z

+
+import "testing"
+
+func TestResolveDashboardAddress(t *testing.T) {


I personally don't think we need this test

Future-Outlier added 2 commits August 5, 2026 17:30

[History Server] Poll Serve, placement group, and Ray Data endpoints …

54251e8

…by default The head collector now polls these endpoints itself every RAY_COLLECTOR_POLL_INTERVAL (default 30s); RAY_COLLECTOR_ADDITIONAL_ENDPOINTS still adds more on top.

[Fix] Use events.NewFakeRecorder in rayservice suspend unit test

34e5a50

RayServiceReconciler.Recorder is events.EventRecorder, so record.NewFakeRecorder does not compile. Every other test in this file already uses events.

Future-Outlier requested review from MortalHappiness, andrewsykim, kevin85421 and rueian as code owners August 5, 2026 22:36

cursor Bot reviewed Aug 5, 2026

View reviewed changes

Comment thread historyserver/pkg/collector/logcollector/runtime/logcollector/collector.go Outdated

Future-Outlier added 3 commits August 5, 2026 17:59

update

df8f4bf

Signed-off-by: Future-Outlier <eric901201@gmail.com>

upd

16f6b8a

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Future-Outlier marked this pull request as draft August 6, 2026 00:32

Future-Outlier marked this pull request as ready for review August 6, 2026 01:25

machichima reviewed Aug 6, 2026

View reviewed changes

Future-Outlier added 2 commits August 6, 2026 12:08

Merge remote-tracking branch 'origin/master' into collector-builtin-e…

010acca

…ndpoints

Future-Outlier force-pushed the collector-builtin-endpoints branch from 0805dee to 17626bc Compare August 6, 2026 17:41

Trigger CI

697ed1c

Signed-off-by: Future-Outlier <eric901201@gmail.com>

machichima reviewed Aug 7, 2026

View reviewed changes

win5923 reviewed Aug 7, 2026

View reviewed changes

Future-Outlier added 3 commits August 8, 2026 12:24

Address endpoint polling review feedback

1092d70

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Merge upstream master into collector-builtin-endpoints

c182988

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Set RayService collector poll interval to 30s

f4f0c84

Signed-off-by: Future-Outlier <eric901201@gmail.com>

machichima approved these changes Aug 9, 2026

View reviewed changes

Comment thread historyserver/config/rayservice.yaml Outdated

Comment thread historyserver/config/rayservice.yaml Outdated

Future-Outlier added 2 commits August 9, 2026 10:10

Address History Server example review feedback

4c7253c

Signed-off-by: Future-Outlier <eric901201@gmail.com>

Align History Server examples on Ray 2.56

3f68297

Signed-off-by: Future-Outlier <eric901201@gmail.com>

win5923 reviewed Aug 9, 2026

View reviewed changes

Comment thread historyserver/config/ray-data.yaml Outdated

Comment thread historyserver/config/rayservice.yaml Outdated

Comment thread historyserver/pkg/collector/logcollector/runtime/logcollector/poll.go Outdated

historyserver: address endpoint polling follow-up review

34802f4

Signed-off-by: Future-Outlier <eric901201@gmail.com>

win5923 approved these changes Aug 9, 2026

View reviewed changes

Comment thread historyserver/config/rayservice.yaml Outdated

Comment thread historyserver/config/ray-data.yaml Outdated

historyserver: configure collector fixtures through env

ea2a85b

Signed-off-by: Future-Outlier <eric901201@gmail.com>

andrewsykim reviewed Aug 10, 2026

View reviewed changes

Future-Outlier added 3 commits August 10, 2026 12:24

Merge remote-tracking branch 'upstream/master' into collector-builtin…

e479b11

…-endpoints # Conflicts: # historyserver/pkg/collector/logcollector/runtime/logcollector/poll.go # historyserver/test/e2e/collector_test.go

historyserver: address default endpoint review

92ae8ed

Signed-off-by: Future-Outlier <eric901201@gmail.com>

historyserver: address default endpoint review feedback

0e42467

Signed-off-by: Future-Outlier <eric901201@gmail.com>

cursor Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread historyserver/pkg/historyserver/router.go

Future-Outlier added 2 commits August 10, 2026 14:16

historyserver: add empty placement group fallback

a1e4983

Signed-off-by: Future-Outlier <eric901201@gmail.com>

historyserver: repair tests after cache changes

8fdb91d

Signed-off-by: Future-Outlier <eric901201@gmail.com>

KunWuLuan approved these changes Aug 11, 2026

View reviewed changes

andrewsykim reviewed Aug 11, 2026

View reviewed changes


		import "testing"

		func TestResolveDashboardAddress(t *testing.T) {

Uh oh!

Conversation

Future-Outlier commented Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Screenshots

Related issue number

Labels

Checks

Manual test instructions

Uh oh!

Uh oh!

Future-Outlier commented Aug 6, 2026

Uh oh!

machichima left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

machichima left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

machichima commented Aug 9, 2026

Uh oh!

machichima left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Future-Outlier commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Future-Outlier commented Aug 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier commented Aug 5, 2026 •

edited

Loading

Future-Outlier commented Aug 9, 2026 •

edited

Loading

Future-Outlier commented Aug 9, 2026 •

edited

Loading

Future-Outlier Aug 10, 2026 •

edited

Loading

Future-Outlier Aug 10, 2026 •

edited

Loading

Future-Outlier Aug 10, 2026 •

edited

Loading