Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

panic: crypto/cipher: invalid buffer overlap #3316

Open
Steven-X97 opened this issue Feb 17, 2025 · 17 comments
Open

panic: crypto/cipher: invalid buffer overlap #3316

Steven-X97 opened this issue Feb 17, 2025 · 17 comments

Comments

@Steven-X97
Copy link

v2ray version:V2Ray 5.10.0
go version: 1.20

I refer to v2ray in a golang project written by myself, which runs on the Debian11 machine. Since February 12, the project has encountered a running exception. The exception log is "panic: runtime error: Slice bounds out of range [: 2239] with capacity of 2048 ", the problem and the same in the #3303, And then found the solution from the #3303 (comment), due to the inconvenience v2ray version upgrade to v5.29.0 directly in the project, So can only refer to [v5.29.0] (https://github.com/v2fly/v2ray-core/releases/tag/v5.29.0) commit the code change, synchronous change their projects v2ray source; The modified project solves the above problem, but as long as you use the Chrome133.x.x.x browser to access the website, an exception will occur, and the exception log is as follows:

panic: crypto/cipher: invalid buffer overlap

goroutine 12830 [running]:
crypto/aes.(*gcmAsm).Seal(0xc002e877c0, {0xc000da2802, 0x0, 0x7fe}, {0xc000965510?, 0x2?, 0xc0021ebc40?}, {0xc000da2800, 0x4e2, 0x800}, ...)
/Go/go1.20/src/crypto/aes/aes_gcm.go:118 +0x4f8
github.com/v2fly/v2ray-core/v5/common/crypto.(*AEADAuthenticator).Seal(0xc001ddf5c0, {0xc000da2802, 0x0, 0x7fe}, {0xc000da2800, 0x4e2, 0x800})
/go/pkg/mod/github.com/v2fly/v2ray-core/[email protected]/common/crypto/auth.go:83 +0x12f
github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).seal(0xc002c9b680, {0xc000da2800, 0x4e2, 0x800})
/go/pkg/mod/github.com/v2fly/v2ray-core/[email protected]/common/crypto/auth.go:260 +0x344
github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).writePacket(0xc002c9b680, {0xc003333cf8, 0x3, 0x7fd254939301?})
/go/pkg/mod/github.com/v2fly/v2ray-core/[email protected]/common/crypto/auth.go:326 +0x233
github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).WriteMultiBuffer(0xc0015e8ab0?, {0xc003333cf8?, 0xc0015e8ab0?, 0xe79c20?})
/go/pkg/mod/github.com/v2fly/v2ray-core/[email protected]/common/crypto/auth.go:353 +0x5f
github.com/v2fly/v2ray-core/v5/common/buf.CopyOnceTimeout({0x191e1a0?, 0xc0015e8ab0?}, {0x191f4c0, 0xc002c9b680}, 0xc0021ebe98?)
/go/pkg/mod/github.com/v2fly/v2ray-core/[email protected]/common/buf/copy.go:122 +0x8e
github.com/v2fly/v2ray-core/v5/proxy/vmess/outbound.(*Handler).Process.func2()
/go/pkg/mod/github.com/v2fly/v2ray-core/[email protected]/proxy/vmess/outbound/outbound.go:156 +0x5b4
github.com/v2fly/v2ray-core/v5/common/task.Run.func1(0xc000410d20?)
/go/pkg/mod/github.com/v2fly/v2ray-core/[email protected]/common/task/task.go:28 +0x2e
created by github.com/v2fly/v2ray-core/v5/common/task.Run
/go/pkg/mod/github.com/v2fly/v2ray-core/[email protected]/common/task/task.go:27 +0xde

@lucfiar
Copy link

lucfiar commented Feb 18, 2025

save here with version 5.29.0

V2Ray 5.29.0 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.23.6 linux/amd64)
A unified platform for anti-censorship.
panic: crypto/cipher: invalid buffer overlap
goroutine 48861 [running]:
crypto/aes.(*gcmAsm).Seal(0xc002568000, {0xc000400002, 0x0, 0x7fe}, {0xc002b48370?, 0xc0020341e0?, 0xc0020d1a98?},>
        crypto/aes/aes_gcm.go:118 +0x4c5
github.com/v2fly/v2ray-core/v5/common/crypto.(*AEADAuthenticator).Seal(0xc0010c73a0, {0xc000400002, 0x0, 0x7fe}, {>
        github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:83 +0x127
github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).seal(0xc0012774f0, {0xc000400000, 0x766, 0x80>
        github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:266 +0x323
github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).writeStream(0xc0012774f0, {0xc00200ec48, 0x3,>
        github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:304 +0x3ea
github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).WriteMultiBuffer(0x1?, {0xc00200ec48?, 0xc003>
        github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:356 +0x50
github.com/v2fly/v2ray-core/v5/common/buf.copyInternal({0x193fd20, 0xc003207610}, {0x1941220, 0xc0012774f0}, 0xc00>
        github.com/v2fly/v2ray-core/v5/common/buf/copy.go:87 +0x15d
github.com/v2fly/v2ray-core/v5/common/buf.Copy({0x193fd20, 0xc003207610}, {0x1941220, 0xc0012774f0}, {0xc0020d1e30>
        github.com/v2fly/v2ray-core/v5/common/buf/copy.go:104 +0x98
github.com/v2fly/v2ray-core/v5/proxy/vmess/inbound.transferResponse({0x1940da0, 0xc0012a1b60}, 0xc000457650, 0xc00>
        github.com/v2fly/v2ray-core/v5/proxy/vmess/inbound/inbound.go:203 +0x226
github.com/v2fly/v2ray-core/v5/proxy/vmess/inbound.(*Handler).Process.func2()
        github.com/v2fly/v2ray-core/v5/proxy/vmess/inbound/inbound.go:312 +0x2c9
github.com/v2fly/v2ray-core/v5/common/task.Run.func1(0xc0024961e0?)
        github.com/v2fly/v2ray-core/v5/common/task/task.go:28 +0x28
created by github.com/v2fly/v2ray-core/v5/common/task.Run in goroutine 48858
        github.com/v2fly/v2ray-core/v5/common/task/task.go:27 +0xdb


@xiaokangwang
Copy link
Contributor

xiaokangwang commented Feb 18, 2025

Would you mind sharing your configuration files for both client and server? It seems the issue is caused by https://github.com/v2fly/v2ray-core/blob/master/common/crypto/auth.go#L266 .

@selboo
Copy link

selboo commented Feb 19, 2025

服务器错误

Feb 19 04:35:13 localhost v2ray[973731]: V2Ray 5.29.0 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.23.6 linux/amd64)
Feb 19 04:35:13 localhost v2ray[973731]: A unified platform for anti-censorship.
Feb 19 09:38:44 localhost v2ray[973731]: panic: crypto/cipher: invalid buffer overlap
Feb 19 09:38:44 localhost v2ray[973731]: goroutine 62089 [running]:
Feb 19 09:38:44 localhost v2ray[973731]: crypto/aes.(*gcmAsm).Seal(0xc000952b40, {0xc00078c802, 0x0, 0x7fe}, {0xc00b070f70?, 0x2?, 0xc0007c5b80?}, {0xc>
Feb 19 09:38:44 localhost v2ray[973731]:         crypto/aes/aes_gcm.go:118 +0x4c5
Feb 19 09:38:44 localhost v2ray[973731]: github.com/v2fly/v2ray-core/v5/common/crypto.(*AEADAuthenticator).Seal(0xc000d7a5c0, {0xc00078c802, 0x0, 0x7fe>
Feb 19 09:38:44 localhost v2ray[973731]:         github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:83 +0x127
Feb 19 09:38:44 localhost v2ray[973731]: github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).seal(0xc00095a6e0, {0xc00078c800, 0x7ae, >
Feb 19 09:38:44 localhost v2ray[973731]:         github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:266 +0x323
Feb 19 09:38:44 localhost v2ray[973731]: github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).writeStream(0xc00095a6e0, {0xc00005dc90, >
Feb 19 09:38:44 localhost v2ray[973731]:         github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:304 +0x3ea
Feb 19 09:38:44 localhost v2ray[973731]: github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).WriteMultiBuffer(0xc000c70230?, {0xc00005>
Feb 19 09:38:44 localhost v2ray[973731]:         github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:356 +0x50
Feb 19 09:38:44 localhost v2ray[973731]: github.com/v2fly/v2ray-core/v5/proxy/vmess/inbound.transferResponse({0x1940da0, 0xc000c63170}, 0xc000c70230, 0>
Feb 19 09:38:44 localhost v2ray[973731]:         github.com/v2fly/v2ray-core/v5/proxy/vmess/inbound/inbound.go:194 +0x1a2
Feb 19 09:38:44 localhost v2ray[973731]: github.com/v2fly/v2ray-core/v5/proxy/vmess/inbound.(*Handler).Process.func2()
Feb 19 09:38:44 localhost v2ray[973731]:         github.com/v2fly/v2ray-core/v5/proxy/vmess/inbound/inbound.go:312 +0x2c9
Feb 19 09:38:44 localhost v2ray[973731]: github.com/v2fly/v2ray-core/v5/common/task.Run.func1(0xc000590690?)
Feb 19 09:38:44 localhost v2ray[973731]:         github.com/v2fly/v2ray-core/v5/common/task/task.go:28 +0x28
Feb 19 09:38:44 localhost v2ray[973731]: created by github.com/v2fly/v2ray-core/v5/common/task.Run in goroutine 62086
Feb 19 09:38:44 localhost v2ray[973731]:         github.com/v2fly/v2ray-core/v5/common/task/task.go:27 +0xdb
Feb 19 09:38:44 localhost systemd[1]: v2ray.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 19 09:38:44 localhost systemd[1]: v2ray.service: Failed with result 'exit-code'.

服务器配置

{
    "log": {
        "access": "none",
        "error": "/var/log/v2ray/error.log",
        "loglevel": "warning"
    },
    "inbounds": [
        {
            "port": xxxxx,
            "listen": "0.0.0.0",
            "tag": "socks-inbound",
            "protocol": "vmess",
            "settings": {
                "clients": [
                    {
                        "id": "xxxx-xxx-xxx-xxx-xxxx",
                        "alterId": 4,
                        "security": "aes-128-gcm"
                    }
                ]
            },
            "sniffing": {
                "enabled": true,
                "destOverride": [
                    "http",
                    "tls"
                ]
            }
        }
    ],
    "outbounds": [
        {
            "protocol": "freedom",
            "settings": {},
            "tag": "direct"
        },
        {
            "protocol": "blackhole",
            "settings": {},
            "tag": "blocked"
        }
    ],
    "routing": {
        "domainStrategy": "IPOnDemand",
        "rules": [
            {
                "type": "field",
                "ip": [
                    "geoip:private"
                ],
                "outboundTag": "blocked"
            },
            {
                "type": "field",
                "domain": [
                    "geosite:category-ads"
                ],
                "outboundTag": "blocked"
            }
        ]
    },
    "dns": {
        "hosts": {
            "domain:v2fly.org": "www.vicemc.net",
            "domain:github.io": "pages.github.com",
            "domain:wikipedia.org": "www.wikimedia.org",
            "domain:shadowsocks.org": "electronicsrealm.com"
        },
        "servers": [
            "1.1.1.1",
            {
                "address": "114.114.114.114",
                "port": 53,
                "domains": [
                    "geosite:cn"
                ]
            },
            "8.8.8.8",
            "localhost"
        ]
    },
    "policy": {
        "levels": {
            "0": {
                "uplinkOnly": 0,
                "downlinkOnly": 0
            }
        },
        "system": {
            "statsInboundUplink": false,
            "statsInboundDownlink": false,
            "statsOutboundUplink": false,
            "statsOutboundDownlink": false
        }
    },
    "other": {}
}

客户端有很多, 以下只是其中一个, 不一定是这个客户端引起的

{
    "outbounds": [
        {
            "mux": {
                "enabled": false
            },
            "protocol": "vmess",
            "settings": {
                "vnext": [
                    {
                        "port": xxxxxx,
                        "users": [
                            {
                                "encryption": "none",
                                "id": "xxxx-xxxx-xxxx-xxxx-xxxx",
                                "level": 0,
                                "security": "none"
                            }
                        ],
                        "address": "server ip "
                    }
                ]
            },
            "streamSettings": {
                "network": "tcp",
                "rawSettings": {
                    "header": {
                        "type": "none"
                    }
                },
                "sockopt": {
                    "mark": 255
                }
            },
            "tag": "QZKzgOJV:xxxxx"
        },
        {
            "streamSettings": {
                "sockopt": {
                    "mark": 255
                }
            },
            "settings": {
                "domainStrategy": "UseIP"
            },
            "protocol": "freedom",
            "tag": "direct"
        },
        {
            "protocol": "blackhole",
            "tag": "blackhole"
        }
    ],

@xiaokangwang

@Steven-X97
Copy link
Author

Would you mind sharing your configuration files for both client and server? It seems the issue is caused by https://github.com/v2fly/v2ray-core/blob/master/common/crypto/auth.go#L266 .

Yes, I read the log analysis, it is the problem in this place, but do not know how to fix it; There is no change in the configuration of my client and server, which was normal before use, and is normal now with other browsers, but this problem will definitely occur as long as I use Chrome. I checked that my Chrome browser was automatically updated to version 133 two days ago. Suspicion may have something to do with the Chrome update.

@Vigilans
Copy link
Contributor

Vigilans commented Feb 19, 2025

I also encountered this issue in recent days (prior to v5.29.0 released):

panic: crypto/cipher: invalid buffer overlap

goroutine 25 [running]:
crypto/aes.(*gcmAsm).Seal(0xc0001368c0, {0xc001567012, 0x0, 0x7ee}, {0xc002280140?, 0xc001567000?, 0x0?}, {0xc001567000, 0x168, 0x800}, ...)
        crypto/aes/aes_gcm.go:118 +0x4c5
github.com/v2fly/v2ray-core/v5/common/crypto.(*AEADAuthenticator).Seal(0xc00b3f0240, {0xc001567012, 0x0, 0x7ee}, {0xc001567000, 0x168, 0x800})
        github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:83 +0x127
github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).seal(0xc00301f1d0, {0xc001567000, 0x168, 0x800})
        github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:266 +0x323
github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).writeStream(0xc00301f1d0, {0xc0011c33e8, 0x1, 0x1})
        github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:304 +0x3eb
github.com/v2fly/v2ray-core/v5/common/crypto.(*AuthenticationWriter).WriteMultiBuffer(0x1?, {0xc0011c33e8?, 0xc002cf43f8?, 0xc000fe1de0?})
        github.com/v2fly/v2ray-core/v5/common/crypto/auth.go:356 +0x50
github.com/v2fly/v2ray-core/v5/common/buf.copyInternal({0x179d100, 0xc002cf4228}, {0x179e560, 0xc00301f1d0}, 0xc00b4f31d0)
        github.com/v2fly/v2ray-core/v5/common/buf/copy.go:87 +0x164
github.com/v2fly/v2ray-core/v5/common/buf.Copy({0x179d100, 0xc002cf4228}, {0x179e560, 0xc00301f1d0}, {0xc000fe1f48, 0x1, 0xc?})
        github.com/v2fly/v2ray-core/v5/common/buf/copy.go:104 +0x98
github.com/v2fly/v2ray-core/v5/proxy/shadowsocks.(*Server).handleConnection.func1()
        github.com/v2fly/v2ray-core/v5/proxy/shadowsocks/server.go:246 +0x3df
github.com/v2fly/v2ray-core/v5/common/task.Run.func1(0x11?)
        github.com/v2fly/v2ray-core/v5/common/task/task.go:28 +0x28
created by github.com/v2fly/v2ray-core/v5/common/task.Run in goroutine 22
        github.com/v2fly/v2ray-core/v5/common/task/task.go:27 +0xdb

My current v2ray-core branch is based on v5.13.0 upstream tag.

@xiaokangwang
Copy link
Contributor

Please try again with https://github.com/v2fly/v2ray-core/releases/tag/v5.29.1 see to if this issue has been fixed.

@1260zqs
Copy link

1260zqs commented Feb 23, 2025

in 5.29.1 this issue still exist

025/02/23 17:16:57.574 [I] [io.go:431] panic: crypto/cipher: invalid buffer overlap
....
2025/02/23 17:16:57.574 [I] [io.go:431] github.com/v2fly/v2ray-core/v5/common/task/task.go:28 +0x28
2025/02/23 17:16:57.574 [I] [io.go:431] created by github.com/v2fly/v2ray-core/v5/common/task.Run in goroutine 4768
2025/02/23 17:16:57.574 [I] [io.go:431] github.com/v2fly/v2ray-core/v5/common/task/task.go
2025/02/23 17:16:57.574 [I] [io.go:431] :27 +0xdb
2025/02/23 17:16:57.584 [W] [asm_amd64.s:1700] v2ray-core: exit status 2
2025/02/23 17:18:26.068 [I] [io.go:431] V2Ray 5.29.1 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.23.6 linux/amd64)
2025/02/23 17:18:26.068 [I] [io.go:431] A unified platform for anti-censorship.

@Vigilans
Copy link
Contributor

in 5.29.1 this issue still exist

025/02/23 17:16:57.574 [I] [io.go:431] panic: crypto/cipher: invalid buffer overlap
....
2025/02/23 17:16:57.574 [I] [io.go:431] github.com/v2fly/v2ray-core/v5/common/task/task.go:28 +0x28
2025/02/23 17:16:57.574 [I] [io.go:431] created by github.com/v2fly/v2ray-core/v5/common/task.Run in goroutine 4768
2025/02/23 17:16:57.574 [I] [io.go:431] github.com/v2fly/v2ray-core/v5/common/task/task.go
2025/02/23 17:16:57.574 [I] [io.go:431] :27 +0xdb
2025/02/23 17:16:57.584 [W] [asm_amd64.s:1700] v2ray-core: exit status 2
2025/02/23 17:18:26.068 [I] [io.go:431] V2Ray 5.29.1 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.23.6 linux/amd64)
2025/02/23 17:18:26.068 [I] [io.go:431] A unified platform for anti-censorship.

Can you paste the full stacktrace for it?

@1260zqs
Copy link

1260zqs commented Feb 23, 2025

in 5.29.1 this issue still exist

025/02/23 17:16:57.574 [I] [io.go:431] panic: crypto/cipher: invalid buffer overlap
....
2025/02/23 17:16:57.574 [I] [io.go:431] github.com/v2fly/v2ray-core/v5/common/task/task.go:28 +0x28
2025/02/23 17:16:57.574 [I] [io.go:431] created by github.com/v2fly/v2ray-core/v5/common/task.Run in goroutine 4768
2025/02/23 17:16:57.574 [I] [io.go:431] github.com/v2fly/v2ray-core/v5/common/task/task.go
2025/02/23 17:16:57.574 [I] [io.go:431] :27 +0xdb
2025/02/23 17:16:57.584 [W] [asm_amd64.s:1700] v2ray-core: exit status 2
2025/02/23 17:18:26.068 [I] [io.go:431] V2Ray 5.29.1 (V2Fly, a community-driven edition of V2Ray.) Custom (go1.23.6 linux/amd64)
2025/02/23 17:18:26.068 [I] [io.go:431] A unified platform for anti-censorship.

Can you paste the full stacktrace for it?

Image

@1260zqs
Copy link

1260zqs commented Feb 23, 2025

there is my server config in gui
Image

@mkondratev
Copy link

Version 5.0.7 works stably.

@xiaokangwang
Copy link
Contributor

@Vigilans I have a guess of what went wrong:
at

cryptoData := bytespool.Alloc(int32(len(b)))
defer bytespool.Free(cryptoData)

The quic sniffer allocate a buffer(%1) and deferred its release(release %1).

bytespool.Free(cryptoData)
cryptoData = newCryptoData

When its size is not sufficient, the cryptoData buffer is released early, and its value is overwrite with a new buffer(%2)

When the function ends, the deferred release %1 is executed again.

as a result, buffer %1 get released twice, and buffer %2 get released 0 time.

The key issue here is that when a function call get deferred, its argument may or may not get a snapshot of the the value passed in: https://go.dev/play/p/F_--epRl2yG .

If my assessment is correct, I would like to request you rewrite all the code in the quic pull request where the argument of the deferred value get modified:

cryptoData = newCryptoData

(the argument passed in was deferred for release in its call stack)

*b = *buf.NewWithSize(cacheLen)

If you do not have the time to rewrite them all I could take over from here.

@Vigilans
Copy link
Contributor

Vigilans commented Feb 26, 2025

@xiaokangwang For reference:

https://stackoverflow.com/questions/41471236/data-passed-by-reference-to-defer

So cryptoData is just copy-by value on deferring...that also means #3320 does not help in terms of buffer release issue because cache is properly released actually.

According to your investigation, we may be able to observe buffer's release status and find out the root cause with some logging. I don't have time to rewrite the code until this weekend.

Possible solutions:

  1. Call defer on each reallocation of buffer
  2. Change defer to something like defer func() { fmt.Printf("%#v\n", user) }()

@xiaokangwang
Copy link
Contributor

@Vigilans I believe #3320 might have helped, considering that the copy could be a shallow copy. So when the argument is a pointer to a struct, when it field in that struct changes, the new value would be received by the deferred call. On the other hand, when the argument is a value itself, like a []byte or int, then the value itself would not be changed.

I think in both case we should just use buf to allocate a buffer sufficiently big to avoid the need to reallocate it, and return an error if that is not sufficient. Since the buffer is in the pool, there won't be too much memory waste about this.

@dyhkwong
Copy link
Contributor

If that is the case,

changing

defer payload.Release()
to

defer func() {
	payload.Release()
}()

and

defer bytespool.Free(cryptoData)
to

defer func() {
	bytespool.Free(cryptoData)
}()

should be enough to prevent the crash. Why not let the affected users do some testing to investigate the root cause before making a real fix?

@xiaokangwang
Copy link
Contributor

should be enough to prevent the crash. Why not let the affected users do some testing to investigate the root cause before making a real fix?

I think your suggestion makes sense. I will write a temporary fix first and let everyone try it out.

@xiaokangwang
Copy link
Contributor

I have released a new version with a temporary fix for this issue, please have a try: https://github.com/v2fly/v2ray-core/releases/tag/v5.29.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants