Add timeout functionality when matching connections

[alyshanjahani-crl]

An issue in the cockroachdb repo (cockroachdb/cockroach#143785) uncovered an undesired behaviour in the goroutines handled by cmux.

Particularly, cmux can hang forever on trying to match a connection.

This can be resolved by users of cmux as illustrated in cockroachdb/cockroach#144170. However, it is an unorthodox solution.

This commit adds functionality to cmux to optionally specify a timeout on matching (reading) connections.

[cthumuluru-crdb]

Can you add a comment on how IO errors are handled?

[alyshanjahani-crl]

If the caller configures a timeout and it occurs that would mean cmux does not match the connect (i.e. it results in an ErrNotMatched)

So i've added comments to that error struct. Hopefully that is intuitive, but lmk if you'd prefer something else.

[cthumuluru-crdb]

Do you mind breaking this into two tests or convert it to table driven tests? I think that makes it easy to follow.

[alyshanjahani-crl]

Made the first test a bit smaller now that we can check the ErrorHandler directly.

Separated the two tests into subtests as well.

[cthumuluru-crdb]

I added a couple of comments. Can you please take a look? Otherwise the changes look good.

[alyshanjahani-crl]

Discussed offline w/ @cthumuluru-crdb

Ideally, an error due to a timeout would be identified as so, i.e. in the implementation for ErrNotMatched the .Timeout() method would indicate if the connection did not match b/c of the configured timeout.

However, this is tricky b/c the interface for the Matcher swallows the error

// Matcher matches a connection based on its content.
type Matcher func(io.Reader) bool

And if we change that type, we have to change consumers of cmux, for example: https://github.com/cockroachdb/cockroach/blob/master/pkg/server/start_listen.go#L123

In this PR we explored using a timer that we could check after not matching in serve, but that has the drawback of extra allocations on every serve of a connection, as well as potentially misidentifying a read error from the matcher as a timeout.

Similarly, if we tried reading from the connection again to determine if the timeout expired (read deadline was exceeded) after not matching, then we might end up blocking on that read for the timeout duration.

So we have settled on implementing the timeout functionality but not identifying the timeout errors.
At least it still makes sense that the consumer receives an ErrNotMatched when a timeout occurs, but again, ideally the error would indicate .Timeout() appropriately.

[cthumuluru-crdb]

LGTM. Thank you!