Skip to content

Commit b5394eb

Browse files
committed
feat(hook): Add contract for lifecycle events
1 parent 0c10b50 commit b5394eb

12 files changed

Lines changed: 722 additions & 1 deletion

File tree

Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ GOIMPORTS_VERSION ?= v0.33.0
3636
# (the out_dir convention in tool/proto/BUILD.bazel) and copied back here. A
3737
# package may hold multiple .proto files (e.g. an RPC contract plus messagequeue
3838
# contracts); all generated stubs land in the same protopb/ dir.
39-
PROTO_PACKAGES = api/base/change api/base/mergestrategy api/base/messagequeue api/runway/messagequeue api/runway api/submitqueue/gateway api/submitqueue/orchestrator api/stovepipe stovepipe/core/messagequeue
39+
PROTO_PACKAGES = api/base/change api/base/hook api/base/mergestrategy api/base/messagequeue api/runway/messagequeue api/runway api/submitqueue/gateway api/submitqueue/orchestrator api/stovepipe stovepipe/core/messagequeue
4040

4141
# Set REPO_ROOT for docker-compose
4242
export REPO_ROOT := $(shell pwd)

api/base/hook/BUILD.bazel

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
load("@rules_go//go:def.bzl", "go_library", "go_test")
2+
3+
go_library(
4+
name = "go_default_library",
5+
srcs = [
6+
"event.go",
7+
"hook.go",
8+
"topics.go",
9+
],
10+
importpath = "github.com/uber/submitqueue/api/base/hook",
11+
visibility = ["//visibility:public"],
12+
deps = [
13+
"//api/base/hook/protopb:go_default_library",
14+
"//api/base/messagequeue/protopb:go_default_library",
15+
"//platform/consumer:go_default_library",
16+
"@org_golang_google_protobuf//encoding/protojson:go_default_library",
17+
"@org_golang_google_protobuf//proto:go_default_library",
18+
],
19+
)
20+
21+
go_test(
22+
name = "go_default_test",
23+
srcs = ["hook_test.go"],
24+
embed = [":go_default_library"],
25+
deps = [
26+
"@com_github_stretchr_testify//assert:go_default_library",
27+
"@com_github_stretchr_testify//require:go_default_library",
28+
"@org_golang_google_protobuf//proto:go_default_library",
29+
"@org_golang_google_protobuf//types/known/structpb:go_default_library",
30+
],
31+
)

api/base/hook/README.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Hook event contract
2+
3+
The published, language-neutral contract for hook events: fire-and-forget lifecycle notifications that let integrations react to a pipeline transition without being able to stall or fail the pipeline. See [the hooks framework RFC](../../../doc/rfc/hook-framework.md) for the design and [the message queue contract RFC](../../../doc/rfc/messagequeue-contract.md) for the conventions it follows.
4+
5+
It lives under `api/base/` rather than `api/{domain}/` because no domain owns it. Every domain publishes this same shape to its own hook topic, so a sink consuming several domains reads one schema rather than one per producer.
6+
7+
Payloads are defined as proto3 messages in [`proto/hook.proto`](proto/hook.proto) and generated into [`protopb/`](protopb); the proto is the authority and a non-Go client compiles against it directly. On the wire, payloads are serialized as protobuf JSON (`protojson`), so the queue keeps storing self-describing JSON. The Go helpers here are generic `protojson` glue — `Marshal(m)` and `Unmarshal[T](b, m)` — plus the two rules that must be identical across producers: how an id is minted and what makes an event well-formed. Field names stay snake_case (`UseProtoNames`) and `int64` fields serialize as JSON strings.
8+
9+
## The envelope
10+
11+
`HookEvent` carries `id`, `source`, `type`, `timestamp_ms`, `version`, and `payload`. The envelope holds only what every consumer keys on uniformly; everything specific to what happened lives in the payload.
12+
13+
`source` and `type` are open strings rather than enums, and `payload` is a `google.protobuf.Struct` rather than a `oneof`. That is the central trade: a producer adds a new event type by publishing it, instead of by changing the wire contract and redeploying every consumer. protojson rejects unknown *enum* values, so an enum here would break existing consumers on every addition.
14+
15+
Subject, queue, and error are deliberately **not** on the envelope. They are facts about a particular occurrence, so they belong in the payload — no major event platform carries a top-level error either.
16+
17+
## Identity and idempotency
18+
19+
`id` is derived from the transition, not random: `source`, `type`, the subject's id, and the subject's post-transition `version`, joined. `NewEventID` mints it. Replaying the delivery that caused the transition therefore mints the *same* id, which is what lets the queue dedupe the redelivery and lets a hook stay idempotent by keying on it. That derivation is why the framework needs no transactional outbox: the publish rides inside the delivery that performed the state write, and a crash before the ack replays both halves safely.
20+
21+
When a transition is not a versioned write there is no version to distinguish occurrences, so the id of the message that caused it stands in, plus an ordinal when one cause publishes several same-typed events. `NewUnversionedEventID` mints that form.
22+
23+
Consumers never parse an id. It is a dedupe and idempotency key, not a structured field.
24+
25+
## Staleness
26+
27+
`version` is the subject's optimistic-locking version immediately after the transition, and `0` when the transition was not a versioned write. Delivery is at-least-once, so a hook can receive an event describing a transition that has since been superseded; comparing this version against the subject's current version in the store is how it tells the two apart. Timestamps cannot answer that, because the clocks belong to different machines.
28+
29+
A domain with no versioned entities (Runway holds no durable state of its own) publishes `0` throughout. That is the normal mode for such a producer, not a degenerate case.
30+
31+
## Payload
32+
33+
Shaped per `type` by the domain that publishes it, add-only, and documented by that domain. It must carry the subject's id, and it must carry any fact recorded nowhere else — merge step outcomes, build failure detail — because for those the event is the only durable record.
34+
35+
It must **not** be an entity snapshot. A snapshot is stale the moment it is redelivered, it competes with the store as a source of truth, and it drags a domain's schema into a contract shared by every domain. Hooks resolve entities from their stores.
36+
37+
## Topic keys
38+
39+
The binding between a topic key and its payload lives in the message's `topic_keys` option (defined in `api/base/messagequeue`); `TopicKeys` reads it back by reflection. A topic key is a stable logical name, not a concrete wire topic — each implementer maps the key to whatever topic name its broker/queue requires, via `consumer.TopicRegistry` in our Go wiring.
40+
41+
| Message | Direction | Topic key |
42+
|---|---|---|
43+
| `HookEvent` | producing domain → hook dispatcher | `hook` |
44+
45+
The key is per-host: each domain runs its own hook topic and its own dispatcher, so two domains sharing one queue backend must map `hook` to distinct topic names.
46+
47+
## Evolution
48+
49+
Contract changes are additive-only: add new fields; never remove, rename, repurpose, or retype an existing field, and never reuse a field number. protojson ignores unknown fields on read and omits zero-valued fields on write, so a new optional field is backward-compatible in both directions. New event types and new payload keys are not contract changes at all — that is the point of the open envelope.

api/base/hook/event.go

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
// Copyright (c) 2026 Uber Technologies, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package hook
16+
17+
import (
18+
"fmt"
19+
"strconv"
20+
"strings"
21+
)
22+
23+
// Consumers never parse an id, so this is a minting convention rather than a
24+
// wire format. All it must guarantee is that two different transitions cannot
25+
// join to the same string.
26+
const idSeparator = "/"
27+
28+
// NewEventID mints the id of an event describing a versioned state write.
29+
//
30+
// Deriving the id rather than randomizing it is what makes replay safe: the same
31+
// transition mints the same id, so the queue dedupes a redelivery and a hook
32+
// stays idempotent without a publisher-side outbox. version is the subject's
33+
// version immediately after the write, which is what separates two transitions
34+
// of the same subject.
35+
func NewEventID(source, eventType, subjectID string, version int32) string {
36+
return strings.Join([]string{source, eventType, subjectID, strconv.Itoa(int(version))}, idSeparator)
37+
}
38+
39+
// NewUnversionedEventID mints the id of an event whose transition was not a
40+
// versioned write, so no version distinguishes one occurrence from the next.
41+
//
42+
// The causing message's id stands in for the version, being stable across
43+
// redeliveries for the same reason a version is. ordinal separates several
44+
// same-typed events published for one cause; pass 0 when there is only one.
45+
func NewUnversionedEventID(source, eventType, subjectID, causeID string, ordinal int) string {
46+
return strings.Join(
47+
[]string{source, eventType, subjectID, causeID, strconv.Itoa(ordinal)},
48+
idSeparator,
49+
)
50+
}
51+
52+
// Validate reports whether e carries the three envelope fields every consumer
53+
// keys on. The rest cannot be checked generically: version is legitimately 0 for
54+
// an unversioned transition and payload is shaped per type.
55+
//
56+
// Both sides call it — a publisher to catch a malformed event before it reaches
57+
// the queue, a consumer because the producer may not have.
58+
func Validate(e *HookEvent) error {
59+
if e == nil {
60+
return fmt.Errorf("hook event is nil")
61+
}
62+
if e.GetId() == "" {
63+
return fmt.Errorf("hook event has no id")
64+
}
65+
if e.GetSource() == "" {
66+
return fmt.Errorf("hook event %q has no source", e.GetId())
67+
}
68+
if e.GetType() == "" {
69+
return fmt.Errorf("hook event %q has no type", e.GetId())
70+
}
71+
return nil
72+
}

api/base/hook/hook.go

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
// Copyright (c) 2026 Uber Technologies, Inc.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
// Package hook holds the hook event contract: the wire payload every domain
16+
// publishes to its own hook topic for fire-and-forget lifecycle side effects.
17+
package hook
18+
19+
import (
20+
"google.golang.org/protobuf/encoding/protojson"
21+
"google.golang.org/protobuf/proto"
22+
23+
"github.com/uber/submitqueue/api/base/hook/protopb"
24+
basemqpb "github.com/uber/submitqueue/api/base/messagequeue/protopb"
25+
)
26+
27+
// HookEvent aliases the generated binding so callers reference the contract
28+
// through this package rather than protopb.
29+
type HookEvent = protopb.HookEvent
30+
31+
// UseProtoNames keeps JSON field names snake_case, matching the declared
32+
// contract rather than protojson's default lowerCamelCase.
33+
var marshalOpts = protojson.MarshalOptions{UseProtoNames: true}
34+
35+
// DiscardUnknown makes an additive contract change backward-compatible: a field
36+
// this consumer does not know yet is ignored rather than rejected.
37+
var unmarshalOpts = protojson.UnmarshalOptions{DiscardUnknown: true}
38+
39+
// Marshal serializes a contract message to protojson bytes for the queue payload.
40+
func Marshal(m proto.Message) ([]byte, error) {
41+
return marshalOpts.Marshal(m)
42+
}
43+
44+
// Unmarshal deserializes protojson bytes into the contract message m.
45+
func Unmarshal[T proto.Message](b []byte, m T) error {
46+
return unmarshalOpts.Unmarshal(b, m)
47+
}
48+
49+
// TopicKeys returns the logical topic keys bound to a message via the
50+
// topic_keys proto option, or nil if it declares none. These are not wire topic
51+
// names; a caller maps each key to its backend's topic.
52+
func TopicKeys(m proto.Message) []string {
53+
opts := m.ProtoReflect().Descriptor().Options()
54+
if opts == nil {
55+
return nil
56+
}
57+
keys, ok := proto.GetExtension(opts, basemqpb.E_TopicKeys).([]string)
58+
if !ok {
59+
return nil
60+
}
61+
return keys
62+
}

0 commit comments

Comments
 (0)