Add const XDR serialization

[leighmcculloch]

Tip

Reviewing Tips

I would focus a review of this change on two areas:

• Sampling a few of the const::{Type} generated types to get an understanding for what code is being generated.
• The new ConstWriter type and the primitive XDR encoding methods.

Note

Part of a stack of PRs listed here:

• Fixing a few issues with the soroban-sdk's production of specs (type ids, name collisions, aliasing, ...) stellar-protocol#2014

What

Add a const feature giving every generated type an alternative XDR encoding that can be used in const contexts. Add a static-lifetime borrowing const::{Type} counterpart for every generated type that directly or transitively contains heap data.

Why

The owned types require heap allocation to construct, so XDR values can't be built from borrowed slices inside const contexts. The const variant types can be constructed in const contexts.

XDR encoding is used by soroban-sdk to encode contract specs during proc-macro execution. The soroban-sdk will be changing when it encodes contracts specs to during compilation so that the specs can contain information only available during compile.

Known limitations

The const::{Type} types are locked down only for use with borrowed data from static lifetimes, which isn't entirely necessary. An earlier implementation I did in #560 and #562 had these borrowed types as {Type}Ref<'a> where the lifetime was configurable and the type looked like it was useful as a ref type. However, in practice, because of the nature that these ref types get nested within arrays inside other ref types, they are not convertible to and from owned types and not really useful outside of a const static lifetime context. For that reason I've kept the types hyper focused on the immediate use case. If we ever have a use case for a more general ref variant of generated types we can reexplore #560.

The const implementation is a separate implementation because a const implementation cannot use traits, which means it can't stream to a writer. It is not practical to maintain a single implementation that offers the same features of both. So instead there are two implementations and a test ensures they work identically.

I would have really liked to fuzz test the const xdr encoding with the existing xdr encoding, however I don't see a way to do that. We can't convert from owned to const types because of the way the const types are structured. Even if they were ref types with lifetimes we can't generate ref types from arbitrary because there would be nothing to own the values. Open to feedback on this, it's possible a slightly different way of architecting the ref types might make this possible.

Other Changes

1. I slipped a fix in here to the existing VecM, BytesM, StringM type Arbitrary implementations. All three could result in the types exceeding their max lengths. When I wrote the const variants of the types, they didn't have the same bug, and so I needed to fix the other for consistency.

2. I modified the stellar-xdr generate arbitrary cli command so that it can accept entropy from a cargo-fuzz corpus file, which is a really helpful way to inspect and view what XDR values the corpus accepts.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 553 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 550 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

fuzz/README.md:19

• This copyable command disables both leak detection and the RSS limit but does not bound the campaign, so the documented ~10 MB/s intentional leak will eventually exhaust memory. Add a finite run or time limit here as the paragraph below requires.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 550 changed files in this pull request and generated no new comments.

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

fuzz/README.md:19

• This example omits the run/time bound that the paragraph below says is mandatory. Copied literally, cargo-fuzz uses libFuzzer's default unlimited run count, while these Arbitrary implementations intentionally leak, so the process will keep growing until interrupted or OOM. Please include a finite bound; for a one-pass corpus check, mirror make fuzz with -runs=0.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 553 changed files in this pull request and generated 2 comments.