all: default should be to run only fast tests, opt in for slow and complicated tests

[ijt]

At present, running vgo test ./... at the root takes 1m20s and outputs lots of failures if its dependencies have not been set up.

The current way to work around that is to add the -short flag mentioned in CONTRIBUTING.md, but it's easy to forget and seems like the reverse of how things should be arranged. Also, that still doesn't filter out the wire tests, and those take 40s to run on my laptop.

I suggest we use build tags to allow opting in to running progressively larger and more complicated tests. We could use the usual Google test size conventions for these. There would probably need to also be an additional, implicit size of tiny for the many test files that run in under 1s or so. Running only those tests would then be the default that happens when running vgo test ./....

[ijt]

We can decouple the running time from the test dependencies. For example we can have tags like over1s for tests that take over 1 second, along with network, filesystem, and database tags for tests that use those things.

For example, a test taking over 1 second and accessing the network would have this build constraint:

// +build over1s,network

and could be run with

vgo test -tags=over1s,network ./...

Thanks to Chris Lewis for this suggestion.

[ijt]

For tests that need to replay, we can have two versions of them, one with a

// +build record

build constraint and another with

// +build replay

A downside of this approach might be some copy/pasting.

[zombiezen]

(This issue is related to #28 by way of test speed, but this is mostly about local development, not CI.)

[zombiezen]

This needs some design work: we don't want to give too many knobs (go test already has quite a few) and we also don't want to give the wrong knobs. This needs to be considered in conjunction with #128, which has been open for a while.

Such a design needs to consider the flow of:

• An individual contributor who is working on Go Cloud interfaces, who may or may not to regenerate cloud replay logs
• An individual contributor who is working on Wire
• Travis pull request builds
• Travis master builds

[cflewis]

Yes, I think there needs to be a design doc as there are many ways to modify the behavior, but no one clear winner AFAICT, all have trade-offs and those trade offs change depending on the test environment/user.