[multiple repos] Upgrade fxamacker/cbor to v2.8 stream-mode branch

[fxamacker]

Problem

Projects outside onflow using atree, cadence, flow-go-sdk, etc. can have build issues if they depend on a newer version of fxamacker/cbor because we currently use a special branch based on v2.4.

For example, Anchorage Digital reported a build issue because they use a package that uses newer fxamacker/cbor (v2.7.0) and we're using fxamacker/cbor v2.4.0-based branch with extra features.

Resolving this will also give us bug fixes, optimizations, etc. from the newer versions of fxamacker/cbor.

Even if the same program depends on packages using different versions of fxamacker/cbor, the compiled program should only include v2.8-based branch.

To save us time, I released fxamacker/cbor v2.8 on Sunday so we can skip upgrading and testing with v2.7.0.

NOTE: Until we upgrade, additional external projects might run into build issues.

Solution

We can use the updated feature/stream-mode branch at fxamacker/cbor and add more tests.

When upgrading from v2.4 to v2.8, we need to run various tests because v2.5 improved error handling.

We may need to add more tests to one or more repos. This may require looking into how CBOR data is currently handled by onflow repos. See "Caveats" below.

More Details

Previously, I tried updating feature/stream-mode branch from v2.4 to v2.7 and atree smoke tests passed 50+ hours on my desktop when using the update. Also, I used a PoC to see if it avoids build issues (simple example with dependencies using different versions of fxamacker/cbor only included latest version in the compiled executable program.

Status

Everything went smoothly and no coding changes were needed by atree, cadence, flow-go, and flow-go-sdk (except bumping version from v2.4 to v2.8).

Most of the work took place in external repos and other onflow repos (e.g., see Atree v0.10.0 release notes).

Progress

Here, "v2.8s" refers to feature/stream-mode branch at fxamacker/cbor v2.8.0.

• Try PoC locally to see if it resolves build issue.
• Open PR fxamacker/cbor#640 to update branch from v2.4.0 to v2.7.0. (+15,260 −4,371)
• Open PR onflow/atree#542 to bump fxamacker/cbor to feature/stream-mode v2.7.0
• Smoke test onflow/atree with feature/stream-mode v2.7.0 (passed 50+ hours on desktop)
• Confirm the approach also resolves build issue for projects (in progress).
• Release tag fxamacker/cbor v2.8.0 (March 30).
• Open PR fxamacker/cbor#650 to update branch v2.7.0 to v2.8.0. (+2,328 −952)
• Review PR fxamacker/cbor#640 (thanks Bastian for review!)
• Review PR fxamacker/cbor#650 (thanks Bastian for review!)
• PR onflow/atree#542 to use fxamacker/cbor v2.8s
• PR onflow/cadence#3818 to use updated atree and fxamacker/cbor v2.8s
• PR onflow/flow-go#7222 to use updated atree, cadence, and fxamacker/cbor v2.8s
• PR onflow/flow-go#7236 to use updated atree, cadence, and fxamacker/cbor v2.8s for flow-go v0.40
• Run flow-go backward compatibility tests (with v2.8s, atree, & cadence)
  • flow-go testnet backward compatibility tests (100K blocks and 1M blocks)
  • flow-go mainnet backward compatibility tests (100K blocks and 1M blocks)
• Test flow-go v0.4 with v2.8s, atree, and cadence using migrationnet
• Run flow-go network message compatibility tests (with v2.8s, atree, & cadence)
  • Sync with Peter about this, mostly to see if leftover input bytes (if possible) should be ignored vs treated as error. fxamacker/cbor v2.5.0 added cbor.UnmarshalFirst(), so it can be used instead of cbor.Unmarshal() if necessary.
• PR onflow/flow-go-sdk to use updated cadence.
• Update and test onflow/flow-emulator using updated v2.8s and cadence
  • Update to Cadence v1.3.4 flow-emulator#819
• Include in these releases:
  • onflow/atree v0.10.0
  • onflow/cadence v1.3.4
  • onflow/flow-go HCU April 10, 2025 on mainnet26 and upcoming v0.40.1
  • onflow/flow-go-sdk v1.4.1

This comment at atree mentions types of tests that should pass before merging PRs into main branch of atree, etc. Other repos might need to also test backward compatibility of network messages, etc.

Caveats

In most cases, fxamacker/cbor v2.8 (and v2.8s) is backward compatible with v2.4.0.

External projects unrelated to onflow upgraded cbor smoothly over the years, but a few required updating their code when v2.5.0 improved error handling.

Specifically, fxamacker/cbor v2.5.0 had bug fix that improved compliance with RFC 8949 related to Unmarshal() error handling for extraneous data (trailing bytes). So this aspect should be tested for compatibility in repos upgrading from versions older than v2.5.0.

See ⭐ Notable Changes to Review Before Upgrading in v2.5.0 release notes.

UPDATE: My comment at fxamacker/cbor#640 (comment) :

..., cadence doesn't use Unmarshal(). And I don't see code in atree or flow-go where Unmarshal() would be given input data with extraneous bytes.

[fxamacker]

So far, everything went smoothly and no coding changes were needed by atree, cadence, flow-go, and flow-go-sdk (except bumping version from v2.4 to v2.8).

For now, I'm postponing merge of atree PR until other repos are also ready to merge.

Next: Discuss network message tests with Access Node team next week (esp. given the improved error handling in v2.5). For example, this upgrade makes cbor.UnmarshalFirst() available, so it should be used instead of cbor.Unmarshal() if leftover bytes are not considered error (i.e. intentional if they exist).

[fxamacker]

Closing as done because the last PR related to this got merged this week (April 22) 🎉:

• Update to Cadence v1.3.4 flow-emulator#819

cc: @j1010001