Skip to content

Fix multiple versions of msgpack being used #7399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 22, 2025
Merged

Fix multiple versions of msgpack being used #7399

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b57e949
Fix multiple versions of msgpack being used
fxamacker May 12, 2025
3bb6979
Merge branch 'master' into fxamacker/cleanup-msgpack-import
fxamacker May 12, 2025
b51fba2
Merge branch 'master' into fxamacker/cleanup-msgpack-import
fxamacker May 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion engine/common/provider/engine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import (
"fmt"

"github.com/rs/zerolog"
"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we double check the existing version and v4 are encoding into the same bytes data? Do we need some test case to verify?

Copy link
Member Author

@fxamacker fxamacker May 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we double check the existing version and v4 are encoding into the same bytes data? Do we need some test case to verify?

Good question.

I don't see tagged releases (on GitHub with release notes) for msgpack and their CHANGELOG.md doesn't show changes within v4 (no details about v4.x -> 4.y versions).

However, there seems to be a lot of bugfixes and performance/memory improvements between v4.0.4 and v4.3.12 based on git comparison.

Do we need some test case to verify?

This might be tricky (for someone unfamiliar with the code and the data used) because some code is calling Marshal on interface type, e.g. flow.Entity, or any type. Maybe people more familiar with the code and data can add some test cases.

BTW, I noticed in some cases non-test code used different msgpack version than the test code. So this PR made msgpack versions used by non-tests and tests consistent too.


"github.com/onflow/flow-go/engine"
"github.com/onflow/flow-go/engine/common/provider/internal"
Expand Down
2 changes: 1 addition & 1 deletion engine/common/provider/engine_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"

"github.com/onflow/flow-go/engine/common/provider"
"github.com/onflow/flow-go/model/flow"
Expand Down
2 changes: 1 addition & 1 deletion engine/common/requester/engine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (
"time"

"github.com/rs/zerolog"
"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"
"go.uber.org/atomic"

"github.com/onflow/flow-go/engine"
Expand Down
2 changes: 1 addition & 1 deletion engine/common/requester/engine_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"
"go.uber.org/atomic"

module "github.com/onflow/flow-go/module/mock"
Expand Down
2 changes: 1 addition & 1 deletion engine/execution/execution_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"
"go.uber.org/atomic"

execTestutil "github.com/onflow/flow-go/engine/execution/testutil"
Expand Down
1 change: 0 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,6 @@ require (
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.15.0
github.com/stretchr/testify v1.10.0
github.com/vmihailenco/msgpack v4.0.4+incompatible
github.com/vmihailenco/msgpack/v4 v4.3.11
go.opentelemetry.io/otel v1.34.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.31.0
Expand Down
2 changes: 0 additions & 2 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1244,8 +1244,6 @@ github.com/urfave/cli v1.22.2/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtX
github.com/urfave/cli v1.22.10/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/vmihailenco/msgpack v4.0.4+incompatible h1:dSLoQfGFAo3F6OoNhwUmLwVgaUXK79GlxNBwueZn0xI=
github.com/vmihailenco/msgpack v4.0.4+incompatible/go.mod h1:fy3FlTQTDXWkZ7Bh6AcGMlsjHatGryHQYUTf1ShIgkk=
github.com/vmihailenco/msgpack/v4 v4.3.11 h1:Q47CePddpNGNhk4GCnAx9DDtASi2rasatE0cd26cZoE=
github.com/vmihailenco/msgpack/v4 v4.3.11/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4=
github.com/vmihailenco/tagparser v0.1.1 h1:quXMXlA39OCbd2wAdTsGDlK9RkOk6Wuw+x37wVyIuWY=
Expand Down
1 change: 0 additions & 1 deletion insecure/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -275,7 +275,6 @@ require (
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/turbolent/prettier v0.0.0-20220320183459-661cc755135d // indirect
github.com/vmihailenco/msgpack v4.0.4+incompatible // indirect
github.com/vmihailenco/msgpack/v4 v4.3.11 // indirect
github.com/vmihailenco/tagparser v0.1.1 // indirect
github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 // indirect
Expand Down
2 changes: 0 additions & 2 deletions insecure/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1188,8 +1188,6 @@ github.com/urfave/cli v1.22.2/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtX
github.com/urfave/cli v1.22.10/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/vmihailenco/msgpack v4.0.4+incompatible h1:dSLoQfGFAo3F6OoNhwUmLwVgaUXK79GlxNBwueZn0xI=
github.com/vmihailenco/msgpack v4.0.4+incompatible/go.mod h1:fy3FlTQTDXWkZ7Bh6AcGMlsjHatGryHQYUTf1ShIgkk=
github.com/vmihailenco/msgpack/v4 v4.3.11 h1:Q47CePddpNGNhk4GCnAx9DDtASi2rasatE0cd26cZoE=
github.com/vmihailenco/msgpack/v4 v4.3.11/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4=
github.com/vmihailenco/tagparser v0.1.1 h1:quXMXlA39OCbd2wAdTsGDlK9RkOk6Wuw+x37wVyIuWY=
Expand Down
1 change: 0 additions & 1 deletion integration/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -316,7 +316,6 @@ require (
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/turbolent/prettier v0.0.0-20220320183459-661cc755135d // indirect
github.com/vmihailenco/msgpack v4.0.4+incompatible // indirect
github.com/vmihailenco/msgpack/v4 v4.3.12 // indirect
github.com/vmihailenco/tagparser v0.1.2 // indirect
github.com/whyrusleeping/go-keyspace v0.0.0-20160322163242-5b898ac5add1 // indirect
Expand Down
2 changes: 0 additions & 2 deletions integration/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1034,8 +1034,6 @@ github.com/urfave/cli v1.22.10/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60Nt
github.com/urfave/cli/v2 v2.25.5/go.mod h1:GHupkWPMM0M/sj1a2b4wUrWBPzazNrIjouW6fmdJLxc=
github.com/viant/assertly v0.4.8/go.mod h1:aGifi++jvCrUaklKEKT0BU95igDNaqkvz+49uaYMPRU=
github.com/viant/toolbox v0.24.0/go.mod h1:OxMCG57V0PXuIP2HNQrtJf2CjqdmbrOx5EkMILuUhzM=
github.com/vmihailenco/msgpack v4.0.4+incompatible h1:dSLoQfGFAo3F6OoNhwUmLwVgaUXK79GlxNBwueZn0xI=
github.com/vmihailenco/msgpack v4.0.4+incompatible/go.mod h1:fy3FlTQTDXWkZ7Bh6AcGMlsjHatGryHQYUTf1ShIgkk=
github.com/vmihailenco/msgpack/v4 v4.3.12 h1:07s4sz9IReOgdikxLTKNbBdqDMLsjPKXwvCazn8G65U=
github.com/vmihailenco/msgpack/v4 v4.3.12/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4=
github.com/vmihailenco/tagparser v0.1.1/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI=
Expand Down
2 changes: 1 addition & 1 deletion model/encodable/keys.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (

"github.com/ethereum/go-ethereum/rlp"
"github.com/fxamacker/cbor/v2"
"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"

"github.com/onflow/crypto"
)
Expand Down
2 changes: 1 addition & 1 deletion model/flow/identity.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
"github.com/ethereum/go-ethereum/rlp"
"github.com/fxamacker/cbor/v2"
"github.com/onflow/crypto"
"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"
)

// DefaultInitialWeight is the default initial weight for a node identity.
Expand Down
2 changes: 1 addition & 1 deletion model/flow/service_event_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
gocmp "github.com/google/go-cmp/cmp"
"github.com/onflow/crypto"
"github.com/stretchr/testify/require"
"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"
"gotest.tools/assert"

"github.com/onflow/flow-go/model/flow"
Expand Down
2 changes: 1 addition & 1 deletion storage/operation/iterate_bench_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import (
"github.com/onflow/flow-go/utils/merr"

"github.com/stretchr/testify/require"
"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"
)

func BenchmarkFindHighestAtOrBelowByPrefixUsingIterator(t *testing.B) {
Expand Down
2 changes: 1 addition & 1 deletion storage/operation/reads.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import (
"errors"
"fmt"

"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"

"github.com/onflow/flow-go/module/irrecoverable"
"github.com/onflow/flow-go/storage"
Expand Down
2 changes: 1 addition & 1 deletion storage/operation/writes.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"bytes"
"fmt"

"github.com/vmihailenco/msgpack"
"github.com/vmihailenco/msgpack/v4"

"github.com/onflow/flow-go/module/irrecoverable"
"github.com/onflow/flow-go/storage"
Expand Down
Loading