internal/fuzzing, cmd/go: compiling fuzz-tests do not include coverage instrumentation

[holiman]

Go version

go1.24.2 linux/amd64

Output of go env in your module/workspace:

AR='ar'
CC='gcc'
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_ENABLED='1'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
CXX='g++'
GCCGO='gccgo'
GO111MODULE=''
GOAMD64='v1'
GOARCH='amd64'
GOAUTH='netrc'
GOBIN=''
GOCACHE='/home/user/.cache/go-build'
GOCACHEPROG=''
GODEBUG=''
GOENV='/home/user/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFIPS140='off'
GOFLAGS=''
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build114624775=/tmp/go-build -gno-record-gcc-switches'
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMOD='/home/user/go/src/github.com/holiman/goevmlab/go.mod'
GOMODCACHE='/home/user/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/user/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/user/.config/go/telemetry'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.24.2'
GOWORK=''
PKG_CONFIG='pkg-config'

What did you do?

I want to build a fuzztest as a standalone binary, for usage in a docker container. However,
the binary complains that "the test binary was not built with coverage instrumentation, so fuzzing will run without coverage guidance and may be inefficient".

I want to enable the coverage instrumentation. Repro steps, below.

Example test:

package main

import (
	"bytes"
	"crypto/sha256"
	"testing"
)

func FuzzFoobar(f *testing.F) {
	f.Fuzz(func(t *testing.T, data []byte) {
		// Invoke some arbitrary methods
		data = sha256.New().Sum(bytes.ToLower(data))
		if len(data) == 0 {
			panic(1)
		}
	})
}

Running it regularly works fine:

$ go test . -fuzz Foo
fuzz: elapsed: 0s, gathering baseline coverage: 0/66 completed
fuzz: elapsed: 0s, gathering baseline coverage: 66/66 completed, now fuzzing with 8 workers
^Cfuzz: elapsed: 3s, execs: 107197 (39642/sec), new interesting: 8 (total: 74)

Compiling it yields complaint:

$ go test  -c .
$ ./testfuzz.test -test.fuzz=Foo -test.fuzzcachedir=/tmp
warning: the test binary was not built with coverage instrumentation, so fuzzing will run without coverage guidance and may be inefficient
warning: starting with empty corpus
fuzz: elapsed: 0s, execs: 0 (0/sec)

If I try to use the -cover when compiling, it does not help:

$ go test -cover -c .
$ ./testfuzz.test -test.fuzz=Foo -test.fuzzcachedir=/tmp
warning: the test binary was not built with coverage instrumentation, so fuzzing will run without coverage guidance and may be inefficient
warning: starting with empty corpus
fuzz: elapsed: 0s, execs: 0 (0/sec)

However, if I add -coverpkg, then something happens. (This is a bit odd, because without specifying coverpkg, it should target all packages):

$ go test -cover -coverpkg=main,crypto/sha256,bytes,testing -c . && ./testfuzz.test -test.fuzz=Foo -test.fuzzcachedir=/tmp
warning: no packages being tested depend on matches for pattern main
warning: the test binary was not built with coverage instrumentation, so fuzzing will run without coverage guidance and may be inefficient
warning: starting with empty corpus
fuzz: elapsed: 0s, execs: 0 (0/sec)
^Cfuzz: elapsed: 3s, execs: 461777 (173901/sec)
PASS
coverage: 20.1% of statements in main, crypto/sha256, bytes, testing

It still complains about lack of coverage instrumentation, but it also emits coverage data afterwards...

Also, if I specify a too large -coverpkg expression, it leads to some internal crash:

$ go test -cover  -coverpkg=... -c . && ./testfuzz.test -test.fuzz=Foo -test.fuzzcachedir=/tmp
warning: the test binary was not built with coverage instrumentation, so fuzzing will run without coverage guidance and may be inefficient
warning: starting with empty corpus
fuzz: elapsed: 0s, execs: 0 (0/sec)
^Cfuzz: elapsed: 1s, execs: 107855 (103023/sec)
PASS

internal error in coverage meta-data tracking:
encountered bad pkgID: 0  at slot: 488  fnID: 6  numCtrs: 1
list of hard-coded runtime package IDs needs revising.
[see the comment on the 'rtPkgs' var in 
 <goroot>/src/internal/coverage/pkid.go]
registered list:
slot: 0 path='internal/asan' 
slot: 1 path='internal/byteorder' 
slot: 2 path='internal/coverage/rtcov' 
slot: 3 path='internal/cpu'  hard-coded id: -2
slot: 4 path='internal/bytealg'  hard-coded id: -11
slot: 5 path='internal/goarch'  hard-coded id: -3
slot: 6 path='internal/abi'  hard-coded id: -8
slot: 7 path='internal/chacha8rand'  hard-coded id: -6
slot: 8 path='internal/godebugs' 
slot: 9 path='internal/goexperiment'  hard-coded id: -12
slot: 10 path='internal/goos'  hard-coded id: -5
slot: 11 path='internal/msan' 
slot: 12 path='internal/profilerecord' 
slot: 13 path='internal/race' 
slot: 14 path='internal/runtime/atomic'  hard-coded id: -4
slot: 15 path='internal/runtime/exithook' 
slot: 16 path='internal/runtime/math'  hard-coded id: -10
slot: 17 path='internal/runtime/sys'  hard-coded id: -7
slot: 18 path='internal/runtime/maps'  hard-coded id: -9
slot: 19 path='internal/runtime/syscall'  hard-coded id: -13
slot: 20 path='internal/stringslite'  hard-coded id: -14
slot: 21 path='runtime'  hard-coded id: -15
slot: 22 path='cmp' 
slot: 23 path='crypto/internal/boring/sig' 
slot: 24 path='crypto/internal/fips140/alias' 
slot: 25 path='crypto/internal/fips140/subtle' 
slot: 26 path='encoding' 
slot: 27 path='crypto/internal/fips140deps/byteorder' 
slot: 28 path='internal/coverage' 
slot: 29 path='internal/coverage/calloc' 
slot: 30 path='internal/coverage/uleb128' 
slot: 31 path='crypto/internal/fips140deps/cpu' 
slot: 32 path='internal/itoa' 
slot: 33 path='internal/unsafeheader' 
slot: 34 path='math/bits' 
slot: 35 path='math' 
slot: 36 path='crypto/subtle' 
slot: 37 path='internal/reflectlite' 
slot: 38 path='errors' 
slot: 39 path='internal/oserror' 
slot: 40 path='iter' 
slot: 41 path='maps' 
slot: 42 path='math/rand/v2' 
slot: 43 path='slices' 
slot: 44 path='sort' 
slot: 45 path='sync/atomic' 
slot: 46 path='internal/sync' 
slot: 47 path='sync' 
slot: 48 path='internal/bisect' 
slot: 49 path='internal/godebug' 
slot: 50 path='crypto/internal/fips140deps/godebug' 
slot: 51 path='internal/testlog' 
slot: 52 path='io' 
slot: 53 path='crypto/internal/randutil' 
slot: 54 path='hash' 
slot: 55 path='hash/fnv' 
slot: 56 path='math/rand' 
slot: 57 path='syscall' 
slot: 58 path='internal/syscall/execenv' 
slot: 59 path='internal/syscall/unix' 
slot: 60 path='time' 
slot: 61 path='context' 
slot: 62 path='internal/poll' 
slot: 63 path='unicode' 
slot: 64 path='unicode/utf16' 
slot: 65 path='unicode/utf8' 
slot: 66 path='bytes' 
slot: 67 path='path' 
slot: 68 path='io/fs' 
slot: 69 path='internal/filepathlite' 
slot: 70 path='os' 
slot: 71 path='crypto/internal/sysrand' 
slot: 72 path='crypto/internal/entropy' 
slot: 73 path='strconv' 
slot: 74 path='crypto' 
slot: 75 path='reflect' 
slot: 76 path='encoding/binary' 
slot: 77 path='encoding/base64' 
slot: 78 path='internal/fmtsort' 
slot: 79 path='fmt' 
slot: 80 path='internal/coverage/cmerge' 
slot: 81 path='internal/coverage/slicereader' 
slot: 82 path='internal/coverage/slicewriter' 
slot: 83 path='internal/coverage/stringtab' 
slot: 84 path='internal/coverage/decodecounter' 
slot: 85 path='runtime/trace' 
slot: 86 path='strings' 
slot: 87 path='bufio' 
slot: 88 path='crypto/internal/fips140' 
slot: 89 path='crypto/internal/fips140/sha3' 
slot: 90 path='crypto/internal/impl' 
slot: 91 path='crypto/internal/fips140/sha256' 
slot: 92 path='crypto/internal/fips140/sha512' 
slot: 93 path='crypto/internal/fips140/hmac' 
slot: 94 path='crypto/internal/fips140/check' 
slot: 95 path='crypto/internal/fips140/aes' 
slot: 96 path='crypto/internal/fips140/drbg' 
slot: 97 path='crypto/internal/fips140/aes/gcm' 
slot: 98 path='crypto/internal/fips140only' 
slot: 99 path='crypto/cipher' 
slot: 100 path='crypto/internal/boring' 
slot: 101 path='crypto/sha256' 
slot: 102 path='encoding/json' 
slot: 103 path='flag' 
slot: 104 path='internal/coverage/decodemeta' 
slot: 105 path='internal/coverage/encodecounter' 
slot: 106 path='internal/coverage/encodemeta' 
slot: 107 path='internal/sysinfo' 
slot: 108 path='path/filepath' 
slot: 109 path='regexp/syntax' 
slot: 110 path='regexp' 
slot: 111 path='internal/coverage/pods' 
slot: 112 path='runtime/debug' 
slot: 113 path='testing' 
slot: 114 path='text/tabwriter' 
slot: 115 path='internal/coverage/cformat' 
slot: 116 path='internal/coverage/cfile' 
slot: 117 path='runtime/coverage' 
remap table:
from  -4  to  14
from  -7  to  17
from  -9  to  18
from  -15  to  21
from  -11  to  4
from  -3  to  5
from  -6  to  7
from  -12  to  9
from  -5  to  10
from  -10  to  16
from  -13  to  19
from  -14  to  20
from  -2  to  3
from  -8  to  6
coverage: 24.0% of statements in ...

What did you see happen?

Standalone fuzzing-binaries are less efficient than fuzz-tests executed via go test.

What did you expect to see?

I'd like to be able to compile fuzz-tests as standalone binaries, to execute in docker-containers which do not have the go runtime.

[gabyhelp]

Related Issues

• testing: cover profile indicates zero coverage of function under test #48178 (closed)
• cmd/go: go 1.22.0: go test throws errors when processing folders not listed in coverpkg argument #65653 (closed)
• cmd/go: -coverpkg=all is not useful #10271 (closed)
• cmd/go: coverpkg flag should be part of cache key #66504
• cmd/go: test with coverage unexpect fail #43242 (closed)
• [dev.fuzz] cmd/go: hangs in "gathering baseline coverage" #46633 (closed)
• cmd/go: inconsistent coverage when using -coverprofile with packages without tests #70244 (closed)

Related Code Changes

• [dev.fuzz] cmd/go/internal: instrument imports of the test
• [dev.fuzz] internal/fuzz: use coverage instrumentation while fuzzing
• [dev.fuzz] internal/fuzz: move coverage capture closer to function

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[JunyangShao]

@matloob @golang/fuzzing

[rolandshoemaker]

Duplicate of #14565 and #46765.