Skip to content

Commit 6da99a8

Browse files
authored
feat(stovepipe): Make project selection optional (#886)
## Summary **What**: - Allow callers to omit project selection and page through all recorded project results for a commit or validation request. - Preserve explicitly selected project results and keep listing and pagination independent of result status. **Why**: - Remove the need for callers to know project identities before inspecting recorded results. - Reuse generic project listing for richer changed-project status reporting. ## Test Plan - [x] Add unit tests. ## Revert Plan - Revert this PR. No schema migration or irreversible data change is introduced; reverting restores the required project selection. ## Issue Links - [CODEM-601](https://linear.app/uber/issue/CODEM-601/optional-projects-field)
1 parent 52c66ff commit 6da99a8

13 files changed

Lines changed: 491 additions & 68 deletions

File tree

‎api/stovepipe/proto/stovepipe.proto‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -116,7 +116,7 @@ message GetProjectStatusByIDRequest {
116116
string queue = 1;
117117
// Canonical decimal request identifier returned from Ingest.
118118
string request_id = 2;
119-
// Required consumer-defined project IDs whose recorded results are requested.
119+
// Optional consumer-defined project IDs. Empty returns all recorded project results.
120120
repeated string projects = 3;
121121
// Maximum requested projects to return in one page. Zero selects the server default.
122122
int32 page_size = 4;
@@ -130,7 +130,7 @@ message GetProjectStatusByURIRequest {
130130
string queue = 1;
131131
// Exact VCS-agnostic URI of the commit under validation.
132132
string change_uri = 2;
133-
// Required consumer-defined project IDs whose recorded results are requested.
133+
// Optional consumer-defined project IDs. Empty returns all recorded project results.
134134
repeated string projects = 3;
135135
// Maximum requested projects to return in one page. Zero selects the server default.
136136
int32 page_size = 4;

‎api/stovepipe/protopb/stovepipe.pb.go‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎doc/rfc/stovepipe/project-status-api.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ The request and response fields have the following API-level meaning. The publis
1717
| `queue` | Yes | Identifies the queue in which to find the validation request. |
1818
| `request_id` | ID lookup | Canonical positive decimal request ID returned by `Ingest`. |
1919
| `change_uri` | URI lookup | Identifies the exact commit URI under validation. |
20-
| `projects` | Yes | Consumer-defined project IDs whose results are requested. At least one ID is required. |
20+
| `projects` | No | Consumer-defined project IDs whose results are requested. Omitted or empty returns all recorded project results. |
2121
| `page_size` | No | Limits one page of results for the requested projects. |
2222
| `page_token` | No | Continues a previous page of results for the requested projects. |
2323

@@ -29,7 +29,7 @@ The request and response fields have the following API-level meaning. The publis
2929
| `updated_at_ms` | Records, in Unix milliseconds, the newest durable lifecycle or result record represented by the response. |
3030
| `repository_breakage_degree` | Returns the whole-repository result when it is durably recorded. |
3131
| `project_results_complete` | Indicates whether the implementation has durably finished producing results for the supplied project IDs. |
32-
| `projects` | Returns recorded results for requested project IDs. |
32+
| `projects` | Returns recorded results for requested project IDs, or all recorded project results when none are supplied. |
3333
| `next_page_token` | Continues requested-project result pagination when another page exists. |
3434

3535
`request_state` is a stable public lifecycle vocabulary: `accepted`, `processing`, `succeeded`, `failed`, `cancelled`, or `superseded`. Clients must tolerate a future value. A terminal request state does not by itself mean that project results are complete.
@@ -56,25 +56,27 @@ Both lookups are queue-scoped and use the same durable request summary and valid
5656

5757
The request-URI mapping admits one request per `(queue, change_uri)`, so the endpoint returns one authoritative request. Revalidation support must introduce an explicit authoritative-request rule; it must not silently change this lookup's meaning.
5858

59-
Queue, URI, request ID, and project ID comparisons are byte-exact and are limited to 255 bytes. An empty queue, lookup selector, project list, or project ID is invalid. A request cannot contain duplicate project IDs. `page_size=0` selects the default of 50; the maximum is 200. Request IDs use the same canonical positive decimal format as `Ingest` and request-history lookups. Page tokens are opaque and bound to the selected request and requested project list; either endpoint can continue a page for the same queue, request, and projects.
59+
Queue, URI, request ID, and project ID comparisons are byte-exact and are limited to 255 bytes. An empty queue, lookup selector, or individual project ID is invalid. A request cannot contain duplicate project IDs. `page_size=0` selects the default of 50; the maximum is 200. Request IDs use the same canonical positive decimal format as `Ingest` and request-history lookups. Page tokens are opaque and bound to the selected request and requested project list; either endpoint can continue a page for the same queue, request, and projects.
6060

6161
## Project Results and Pagination
6262

63-
The caller supplies the project IDs for which it wants recorded results. The API does not define project discovery, validation selection, or attribution. A project absent from the response has no universal meaning and consumers must not infer that it is green.
63+
The caller may supply the project IDs for which it wants recorded results. The API does not define project discovery, validation selection, or attribution. A project absent from the response has no universal meaning and consumers must not infer that it is green.
6464

65-
The response includes only results for the supplied project IDs, in the supplied order, and paginates that list. An implementation may record results for every requested project, only requested projects with a non-green result, or another documented subset.
65+
When project IDs are supplied, the response includes only results for those IDs, in the supplied order, and paginates that list. An implementation may record results for every requested project, only requested projects with a non-green result, or another documented subset.
66+
67+
With no project IDs supplied, the response lists all recorded project results in project-ID order, excluding the repository fact. Results are a live view; restart pagination to discover facts recorded behind the cursor.
6668

6769
Each project result is an immutable validation fact keyed by queue, commit URI, and project ID. A result must identify the selected request. `project_results_complete` is false until the implementation has durably finished producing results for the supplied project IDs; a completed empty set is valid.
6870

6971
Callers use `project_results_complete`, rather than the presence or absence of an individual project result, to determine whether the implementation has finished producing results for the requested projects. `next_page_token` is empty only on the final page. A terminal lifecycle notification is emitted only after the repository result and any applicable project completion record are durable. If notification delivery fails, consumers can recover the same result through either endpoint.
7072

7173
## Implementation
7274

73-
Both endpoints return durable lifecycle summaries, the whole-repository fact when present, and any recorded facts for requested projects in paginated request order. `project_results_complete` remains false until a durable completion projection is available. Missing facts are omitted; an explicitly recorded zero degree remains present. `updated_at_ms` is the maximum timestamp of the summary and the facts represented on the page.
75+
Both endpoints return durable lifecycle summaries, the whole-repository fact when present, and recorded facts for requested projects in paginated request order, or all recorded project results in project-ID order when none are requested. `project_results_complete` remains false until a durable completion projection is available. Missing facts are omitted; an explicitly recorded zero degree remains present. `updated_at_ms` is the maximum timestamp of the summary and the facts represented on the page.
7476

7577
## Errors and Authorization
7678

77-
- Invalid selectors, an empty project list, an invalid project ID, duplicate project IDs, or malformed page tokens are user errors.
79+
- Invalid selectors, an invalid project ID, duplicate project IDs, or malformed page tokens are user errors.
7880
- An unknown queue-scoped request ID or commit URI is not found.
7981
- A request-URI mapping whose summary is not visible is retryable.
8082
- A selector and summary that disagree, a result belonging to another request, or a completion record with a missing project result is an internal consistency error.

‎stovepipe/controller/project_status.go‎

Lines changed: 135 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ import (
2323
"errors"
2424
"fmt"
2525
"math"
26-
"reflect"
26+
"slices"
2727

2828
"github.com/uber-go/tally"
2929
"github.com/uber/submitqueue/platform/errs"
@@ -39,9 +39,11 @@ const (
3939
)
4040

4141
type projectStatusPageToken struct {
42-
RequestID string `json:"request_id"`
43-
Projects []string `json:"projects"`
44-
NextIndex int `json:"next_index"`
42+
Queue string `json:"queue"`
43+
RequestID string `json:"request_id"`
44+
Projects []string `json:"projects"`
45+
NextIndex int `json:"next_index"`
46+
AfterProject string `json:"after_project,omitempty"`
4547
}
4648

4749
// ProjectStatusNotFoundError indicates that no validation request matches a lookup selector.
@@ -102,7 +104,7 @@ func NewProjectStatusController(logger *zap.SugaredLogger, scope tally.Scope, st
102104
}
103105
}
104106

105-
// GetProjectStatus returns the selected request summary and its repository validation fact, if recorded.
107+
// GetProjectStatus returns the selected request summary and recorded validation facts.
106108
func (c *projectStatusController) GetProjectStatus(ctx context.Context, req entity.GetProjectStatusRequest) (result entity.GetProjectStatusResult, retErr error) {
107109
op := metrics.Begin(c.metricsScope, "get_project_status", metrics.StorageLatencyBuckets, metrics.TagsFromContext(ctx)...)
108110
defer func() { op.Complete(retErr) }()
@@ -158,26 +160,19 @@ func (c *projectStatusController) GetProjectStatus(ctx context.Context, req enti
158160
result.UpdatedAtMs = fact.CreatedAt
159161
}
160162
}
161-
page, err := selectProjectStatusPage(req, requestSummary.RequestID)
163+
164+
page, err := loadProjectStatusPage(ctx, req, requestSummary, factStore)
162165
if err != nil {
163166
return entity.GetProjectStatusResult{}, err
164167
}
165-
for _, project := range page.projects {
166-
projectFact, err := factStore.Get(ctx, requestSummary.URI, project)
167-
if err != nil {
168-
if storage.IsNotFound(err) {
169-
continue
170-
}
171-
return entity.GetProjectStatusResult{}, fmt.Errorf("GetProjectStatus failed to load project fact for request %q and project %q: %w", requestSummary.RequestID, project, err)
172-
}
173-
if err := validateProjectFact(projectFact, requestSummary, project); err != nil {
174-
return entity.GetProjectStatusResult{}, err
175-
}
176-
result.ProjectValidationFacts = append(result.ProjectValidationFacts, projectFact)
168+
169+
result.ProjectValidationFacts = page.facts
170+
for _, projectFact := range page.facts {
177171
if projectFact.CreatedAt > result.UpdatedAtMs {
178172
result.UpdatedAtMs = projectFact.CreatedAt
179173
}
180174
}
175+
181176
result.NextPageToken = page.nextToken
182177

183178
c.logger.Debugw("project status retrieved", "request_id", requestSummary.RequestID, "queue", requestSummary.Queue, "change_uri", requestSummary.URI, "has_repository_result", result.HasRepositoryValidationFact, "project_results", len(result.ProjectValidationFacts))
@@ -200,9 +195,6 @@ func validateProjectStatusRequest(req entity.GetProjectStatusRequest) error {
200195
return fmt.Errorf("GetProjectStatus invalid change_uri=%q: %w", req.ChangeURI, err)
201196
}
202197
}
203-
if len(req.Projects) == 0 {
204-
return fmt.Errorf("GetProjectStatus projects must be non-empty: %w", ErrInvalidRequest)
205-
}
206198
seen := make(map[string]struct{}, len(req.Projects))
207199
for _, project := range req.Projects {
208200
if err := validateHistoryIdentifier("project", project); err != nil {
@@ -220,55 +212,153 @@ func validateProjectStatusRequest(req entity.GetProjectStatusRequest) error {
220212
}
221213

222214
type projectStatusPage struct {
223-
projects []string
215+
facts []entity.ValidationFact
224216
nextToken string
225217
}
226218

227-
func selectProjectStatusPage(req entity.GetProjectStatusRequest, requestID string) (projectStatusPage, error) {
228-
start := 0
229-
if req.PageToken != "" {
230-
token, err := decodeProjectStatusPageToken(req.PageToken)
231-
if err != nil {
232-
return projectStatusPage{}, err
233-
}
234-
if token.RequestID != requestID || !reflect.DeepEqual(token.Projects, req.Projects) || token.NextIndex <= 0 || token.NextIndex >= len(req.Projects) {
235-
return projectStatusPage{}, fmt.Errorf("GetProjectStatus page_token does not match the selected request and projects: %w", ErrInvalidRequest)
236-
}
237-
start = token.NextIndex
219+
func loadProjectStatusPage(
220+
ctx context.Context,
221+
req entity.GetProjectStatusRequest,
222+
summary entity.RequestSummary,
223+
store storage.ValidationFactStore,
224+
) (projectStatusPage, error) {
225+
token, err := parseProjectStatusPageToken(req, summary)
226+
if err != nil {
227+
return projectStatusPage{}, err
238228
}
229+
239230
pageSize := int(req.PageSize)
240231
if pageSize == 0 {
241232
pageSize = defaultProjectStatusPageSize
242233
}
243-
end := min(start+pageSize, len(req.Projects))
244-
page := projectStatusPage{projects: req.Projects[start:end]}
245-
if end < len(req.Projects) {
246-
token, err := encodeProjectStatusPageToken(projectStatusPageToken{RequestID: requestID, Projects: req.Projects, NextIndex: end})
234+
235+
if len(req.Projects) == 0 {
236+
return listProjectStatusPage(ctx, summary, store, token, pageSize)
237+
}
238+
239+
return getRequestedProjectStatusPage(ctx, summary, store, token, pageSize)
240+
}
241+
242+
func getRequestedProjectStatusPage(
243+
ctx context.Context,
244+
summary entity.RequestSummary,
245+
store storage.ValidationFactStore,
246+
token projectStatusPageToken,
247+
pageSize int,
248+
) (projectStatusPage, error) {
249+
end := min(token.NextIndex+pageSize, len(token.Projects))
250+
var page projectStatusPage
251+
252+
for _, project := range token.Projects[token.NextIndex:end] {
253+
fact, err := store.Get(ctx, summary.URI, project)
247254
if err != nil {
248-
return projectStatusPage{}, fmt.Errorf("GetProjectStatus failed to create page token: %w", err)
255+
if storage.IsNotFound(err) {
256+
continue
257+
}
258+
259+
return projectStatusPage{}, fmt.Errorf(
260+
"GetProjectStatus failed to load project fact for request %q and project %q: %w",
261+
summary.RequestID, project, err,
262+
)
263+
}
264+
265+
if err := validateProjectFact(fact, summary, project); err != nil {
266+
return projectStatusPage{}, err
267+
}
268+
269+
page.facts = append(page.facts, fact)
270+
}
271+
272+
var err error
273+
if end < len(token.Projects) {
274+
token.NextIndex = end
275+
page.nextToken, err = encodeProjectStatusPageToken(token)
276+
}
277+
278+
return page, err
279+
}
280+
281+
func listProjectStatusPage(
282+
ctx context.Context,
283+
summary entity.RequestSummary,
284+
store storage.ValidationFactStore,
285+
token projectStatusPageToken,
286+
pageSize int,
287+
) (projectStatusPage, error) {
288+
facts, err := store.ListProjects(ctx, summary.URI, token.AfterProject, pageSize+1)
289+
if err != nil {
290+
return projectStatusPage{}, fmt.Errorf("GetProjectStatus failed to list project facts for request %q: %w", summary.RequestID, err)
291+
}
292+
293+
for _, fact := range facts {
294+
if err := validateProjectFact(fact, summary, fact.Project); err != nil {
295+
return projectStatusPage{}, err
296+
}
297+
298+
if fact.Project == "" {
299+
return projectStatusPage{}, &ProjectStatusConsistencyError{message: "project listing contains a repository fact"}
249300
}
250-
page.nextToken = token
251301
}
252-
return page, nil
302+
303+
page := projectStatusPage{facts: facts}
304+
if len(facts) > pageSize {
305+
page.facts = facts[:pageSize]
306+
token.AfterProject = page.facts[pageSize-1].Project
307+
page.nextToken, err = encodeProjectStatusPageToken(token)
308+
}
309+
310+
return page, err
253311
}
254312

255313
func encodeProjectStatusPageToken(token projectStatusPageToken) (string, error) {
256314
contents, err := json.Marshal(token)
257315
if err != nil {
258-
return "", err
316+
return "", fmt.Errorf("GetProjectStatus failed to create page token: %w", err)
259317
}
318+
260319
return base64.RawURLEncoding.EncodeToString(contents), nil
261320
}
262321

263-
func decodeProjectStatusPageToken(encoded string) (projectStatusPageToken, error) {
264-
contents, err := base64.RawURLEncoding.DecodeString(encoded)
322+
func parseProjectStatusPageToken(
323+
req entity.GetProjectStatusRequest,
324+
summary entity.RequestSummary,
325+
) (projectStatusPageToken, error) {
326+
if req.PageToken == "" {
327+
return projectStatusPageToken{
328+
Queue: summary.Queue,
329+
RequestID: summary.RequestID,
330+
Projects: req.Projects,
331+
}, nil
332+
}
333+
334+
invalidToken := fmt.Errorf("GetProjectStatus page_token does not match the selected request and projects: %w", ErrInvalidRequest)
335+
336+
contents, err := base64.RawURLEncoding.DecodeString(req.PageToken)
265337
if err != nil {
266-
return projectStatusPageToken{}, fmt.Errorf("GetProjectStatus invalid page_token: %w", ErrInvalidRequest)
338+
return projectStatusPageToken{}, invalidToken
267339
}
340+
268341
var token projectStatusPageToken
269342
if err := json.Unmarshal(contents, &token); err != nil {
270-
return projectStatusPageToken{}, fmt.Errorf("GetProjectStatus invalid page_token: %w", ErrInvalidRequest)
343+
return projectStatusPageToken{}, invalidToken
271344
}
345+
346+
if token.Queue != summary.Queue ||
347+
token.RequestID != summary.RequestID ||
348+
!slices.Equal(token.Projects, req.Projects) {
349+
return projectStatusPageToken{}, invalidToken
350+
}
351+
352+
if len(req.Projects) == 0 {
353+
if token.NextIndex != 0 || validateHistoryIdentifier("project", token.AfterProject) != nil {
354+
return projectStatusPageToken{}, invalidToken
355+
}
356+
} else if token.AfterProject != "" ||
357+
token.NextIndex <= 0 ||
358+
token.NextIndex >= len(req.Projects) {
359+
return projectStatusPageToken{}, invalidToken
360+
}
361+
272362
return token, nil
273363
}
274364

0 commit comments

Comments
 (0)