map delete not finding keys when value is largeish, key hash seems to be incorrect

[anuraaga]

I am currently debugging this code that tries to delete the keys in a map (to empty it while retaining bucket storage)

https://github.com/corazawaf/coraza/blob/v3/dev/internal/corazawaf/rulegroup.go#L105

(for the below, note that I tried copying the keys into a slice before deleting to see if it was iterator+delete behavior problem but it was all the same so simplified back to normal map iteration).

I noticed a memory leak and after adding if len(transformationCache) > 0 { panic } after the deletion could verify that the map wasn't getting cleared with tinygo wasm. It works fine with Go.

I have added this instrumentation to hashmap.go

https://github.com/anuraaga/tinygo/compare/custom-gc...anuraaga:tinygo:hashmapdelete-debug?expand=1

And shrunk down the test to similar one as last time

https://github.com/corazawaf/coraza-proxy-wasm/compare/main...anuraaga:coraza-1225?expand=1#diff-173fbfd8d8844658344b121461b4290d0a85230caae9825240705df8130e8b75R33
~/go/bin/tinygo build -scheduler=none -target=wasi -opt=2 -o mapbench.wasm

The debug output looks something like this

hashmapBinaryGet 0x0001019c 0x0000ffd0 16 1666840535
could find key
hashmapBinaryDelete 0x0001019c 0x0000ffb8 16 1666840535
delete could find key 0x0001019c 0x0000ffb8 1666840535
hashmapBinaryGet 0x0001019c 0x0000ffd0 16 1666840535
couldn't find key

The key address for get is 0x0000ffd0 and delete is 0x0000ffb8. That being said the hash is the same in this example so it's being able to clear the map but with the same instrumentation when looking at the output for the original coraza code, the hash values were also different. I'm not sure why I wasn't able to reproduce this hash value difference, but either way, the key is in a local variable k, which there is only one of, so intuitively I suppose those addresses must be the same and the difference is unexpected.

One weird thing I found while making the repro is it seems the value struct needs to be more than 2 strings worth of size to exhibit the behavior - with three fields, get and delete have different addresses, while with two fields they are the same.

Looked through the code in hashmap.go and map.go (compiler) and couldn't see anything suspicious, the code paths for get/lookup vs delete look basically the same for both, but the difference does cause our real-world code to fail with the map not being cleared. With the code looking the same, the issue may be in IR reduction?

Note the above simpleish test case approach is also applied to the real world code here (which is where I was observing the address+key value difference)

https://github.com/corazawaf/coraza/compare/v3/dev...anuraaga:scratch?expand=1

The output looks like this (we can see the different hash values)

[2022-12-28 04:04:56.210312][25][info][wasm] [source/extensions/common/wasm/context.cc:1170] wasm log coraza-filter my_vm_id: hashmapBinaryGet 0x09887b70 0x0000fb98 16 2748313615
[2022-12-28 04:04:56.210312][25][info][wasm] [source/extensions/common/wasm/context.cc:1170] wasm log coraza-filter my_vm_id: could find key
[2022-12-28 04:04:56.210312][25][info][wasm] [source/extensions/common/wasm/context.cc:1170] wasm log coraza-filter my_vm_id: hashmapBinaryDelete 0x09887b70 0x0000fbf8 16 3003228291
[2022-12-28 04:04:56.210313][25][info][wasm] [source/extensions/common/wasm/context.cc:1170] wasm log coraza-filter my_vm_id: delete couldn't find key 0x09887b70
[2022-12-28 04:04:56.210313][25][info][wasm] [source/extensions/common/wasm/context.cc:1170] wasm log coraza-filter my_vm_id: hashmapBinaryGet 0x09887b70 0x0000fb98 16 2748313615
[2022-12-28 04:04:56.210314][25][info][wasm] [source/extensions/common/wasm/context.cc:1170] wasm log coraza-filter my_vm_id: could find key

[anuraaga]

Digging through LLVM docs, I did come to understand that because get and delete create a temporarly alloca, it is conceivable for those to be different and then different addresses (though that would be an unfortunate deoptimization if copying the key struct from execution stack to c stack twice). I guess there is something about the real-world code that is causing the contents of the allocas to be different, which I still wasn't able to repro in the simpler example...

[anuraaga]

For context, changing the map value type to pointer solved the issue, though is not ideal since we didn't want to heap allocate for this cache

corazawaf/coraza#554

This is because it seems like the problematic behavior only happens with larger value size. Or perhaps just coincidence / luck

[dgryski]

I've done a lot of work on the hashing code and the hash maps. I'll take a look at this.

[dgryski]

Hmm... can you come up with a reproducer based on the mapbench code that doesn't require patching the runtime to demonstrate it? (I've patched my runtime but it's much easier if we have something that can if if ... { panic }

[anuraaga]

@dgryski Do you need a small amount of code or a small command that reliably reproduces? I can send the latter easily based on coraza-proxy-wasm if it will help, if the former I'll try some more and see what I can do

[dgryski]

Both are nice. The small amount of code is better but the small command is useful until that arrives.

[anuraaga]

Thanks @dgryski.

You'll need your TinyGo built with #3302 and docker available. ftw will fail after a few requests

git clone https://github.com/anuraaga/coraza-proxy-wasm.git --branch repro-mapdelete
cd coraza-proxy-wasm
LOCAL_TINYGO=1 go run mage.go build
go run mage.go ftw
tail -n 20 build/ftw-envoy.log

[dgryski]

Smaller reproducer:

~/go/src/github.com/dgryski/bug $ cat bug.go
package main

type V struct {
	buf [256]byte
	s1  string
	s2  string
	s3  string
}

func main() {
	m := make(map[int]V)

	m[0] = V{s1: "hello", s2: ",", s3: "world"}

	println(m[0].s1)
	delete(m, 0)
	println(m[0].s1)
}
~/go/src/github.com/dgryski/bug $ go run bug.go
hello

~/go/src/github.com/dgryski/bug $ tinygo run bug.go
hello
hello

Note that if the buf struct is after the strings, then the bug is not present.

[dgryski]

Oh duh. The hashmap code limits value size to uint8 for some reason: https://github.com/tinygo-org/tinygo/blob/release/src/runtime/hashmap.go#L19

That's an easy enough fix.

[dgryski]

#3377

[dgryski]

Annoyingly while this is a bug I'm not certain this actually related to the issue at hand, given that the values in the test program are much smaller.

[dgryski]

I wonder if this is the same memory corruption from the garbage collector missing pointers that we've been seeing elsewhere?