Skip to content

Capture a query's binds as plain values, without a digest or a key - #254

Merged
maverox merged 1 commit into
mainfrom
work/binds-as-plain-values
Sep 25, 2026
Merged

maverox merged 1 commit into
mainfrom
work/binds-as-plain-values

Conversation

@maverox

@maverox maverox commented Sep 25, 2026

Copy link
Copy Markdown
Collaborator

#247 captured a query's binds as structure, keyed by placeholder, but replaced every scalar with a keyed digest and fell back to diesel's debug rendering when no key was installed. So the structure, which is what removes the order-only db divergences, worked only once a secret was installed in every recording and replay deployment and kept equal between them. This drops the digest. Bind values are captured as the plain values they are, and structure is the only behaviour, with no configuration.

Acting on this needs fresh recordings, as #247 did: every db call's args change shape, so it can't be validated against a recording that exists today.

What changes

  • Scalars are plain values, decoded from the bytes diesel sends to Postgres by type:
    • int2, int4 and int8 become numbers; float4 and float8 become numbers (NaN and infinities become their names).
    • bool becomes a boolean.
    • text and varchar become strings, and so do host enums, since Postgres' binary form of an enum is its label.
    • date, timestamp and timestamptz become ISO text, including the infinities.
    • Anything with no faithful JSON form (bytea, numeric, any type not listed) becomes its bytes in Postgres' hex notation, "\x…". It is never guessed at.
  • Documents and arrays are as before. json and jsonb become documents with their own numbers and strings. One-dimensional arrays become arrays in the order bound, with null elements kept in place.
  • Binds stay keyed by placeholder, {"$1": …, "$2": …}, because a bind's position is part of what the statement means. An object is compared and hashed by key, so no rule that forgives array order can make two values traded between placeholders read as the same query.
  • Removed: deja_runtime::capture_key (the HMAC, install_capture_key, the key id), its hmac, sha2 and hex dependencies, and the keyless fallback. CapturedQuery.binds is now always present. A capture that fails still records {"capture_failed": step} and no operand.

What becomes readable, concretely

  • Bind values now reach the args in the clear. For statements that return a row, this adds nothing: in one cycle, 7,142 of the 7,146 binds that diesel's rendering masked were already on the recording unmasked, in the RETURNING row of the same event.
  • The 4 others were binds of inserts that failed with a unique violation, so no row came back:
    • 3 are Encryption values. That is ciphertext, and a tape holds ciphertext either way.
    • 1 is refund.metadata: merchant-supplied JSON, now readable. This is the one to weigh.
  • Host Secret masking can't be honoured from the bytes. A Secret's ToSql writes its inner value, and diesel's only per-bind Debug pass is crate-private.
  • Bytea columns hold encrypted fields and are captured as their hex. The capture is exact, and larger than a digest.

How this meets the scorer's order rule

  • The scorer compares an array that is not all-numeric as a multiset, and keeps an all-numeric array in order (Read order by one identity at the lookup, the diff and the round trip #248).
  • Status sets. The webhook status sets (HashSet<IntentStatus> and the refund, payout and dispute equivalents, as fields of WebhookDetails, bound as json through serde) capture as JSON arrays of strings in the set's iteration order: ["failed", "succeeded"]. That is the multiset shape, so a reordered set is absorbed and named rather than blocking.
    • a_status_set_captures_as_an_array_of_strings_in_either_order pins the shape: two orders, the same members, all strings.
    • This is what made closing hyperswitch#14371 correct. That PR would have ordered these sets in the router, and it was closed because deja tolerates their order.
  • Integer arrays stay numbers in the order bound. Where the digest made every array an array of strings, a reordered all-numeric bind list now blocks instead of being absorbed.
    • None exists in hyperswitch today: every eq_any in its queries is over strings or enums.
    • In one cycle, the 19 reordered bind lists in blocking rows were 16 enum status sets and 3 string lists, none numeric.
    • A numeric reorder in a later cycle would be this interaction, not a regression elsewhere.

Evidence

  • Tests. 17 in deja-diesel/tests/bind_capture.rs:
    • Every scalar type, with dates across the whole range: before 2000, 1900-03-01 and 2100-03-01 (century years that aren't leap years), and both infinities.
    • An integer IN-list kept numeric and in order.
    • The status-set shape, and host enums and arrays of them.
    • Nullable-jsonb arrays, plain json, and nulls.
    • Values traded between placeholders.
    • Both failure steps.
    • The red fixture from Capture a query's binds as structure, with every value digested #247, which asserts that the two debug renderings differ and the captures don't.
  • Mutation. 26 mutants, 26 killed. They cover:
    • Each decoder: every int width, bool, float4, the epoch offset, the leap rule, century years, negative time, infinities, the timestamptz zone, and the hex form.
    • The array rule: by type, element types, and order kept.
    • Placeholder numbering, and binds as a list.
    • Null, and both failure steps.
  • just verify: exit 0, clippy -D warnings with 0 diagnostics, 1,333 Rust tests and 79 web tests passed, 0 failed. It was run before the last four test additions; the crate's own suite was run after them.
  • MSRV. cargo +1.85.0 check passes on the runtime crates.
  • Independent review. It found every decoder consistent with diesel 2.2.10's ToSql and Postgres' binary formats, and no hyperswitch column type decoded wrongly rather than hexed. Its test gaps are covered above.

The structured bind capture replaced every scalar with a keyed digest and fell
back to diesel's debug rendering when no key was installed, so the structure
that removes the order-only db divergences worked only once a secret was
installed in every recording and replay deployment and kept equal between
them. Bind values are now captured as the plain values they are, decoded from
the bytes diesel sends to Postgres by type: numbers as numbers, text and host
enums as strings, dates and timestamps as ISO text, json and jsonb as
documents, arrays as arrays, and anything with no faithful JSON form as its
bytes in hex. Binds stay keyed by placeholder, since a bind's position is part
of what the statement means. The capture key, its id and the keyless fallback
are removed, so structure is the only behaviour and needs no configuration.

For statements that return a row the values were already on the recording
unmasked, in the returned row. Integer arrays now stay numbers in the order
bound, so the scorer keeps their order where it bags other arrays.
@maverox
maverox merged commit 395782a into main Sep 25, 2026
10 checks passed
@maverox maverox self-assigned this Sep 25, 2026
@maverox
maverox deleted the work/binds-as-plain-values branch September 25, 2026 13:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant