Repository navigation
Conversation
e0dd905 to
6911027
Compare
| n, err = io.Copy(dst, src) | ||
|
|
||
| src.Close() | ||
| dst.Close() |
There was a problem hiding this comment.
Closing manually is error prone.
We should extract a function doing streaming and copying and use defer in the function. Then the retry loop can be much simpler.
Closing dst without checking for errors hides failure to flush data to the log. We should check the error and warn in this case.
| // failing to stream it again is not expected. | ||
| a.log.Warnf("Cannot get log for \"%s/%s\": %s", container, which, err) | ||
| return | ||
| } |
There was a problem hiding this comment.
Why do we need special handling based on number of attempts? If we treat Stream error as fatal error, it does not matter if this is the first or second attempt.
The only reason may be if we log a different message about partial log fetched by the previous attempt.
There was a problem hiding this comment.
These tests use too much mocking - they may succeed when real log streaming will fail and increase maintenance cost. We should test this using real cluster if we can trigger the goway error.
| } | ||
|
|
||
| if attempt < logStreamAttempts && isGoAwayError(err) { | ||
| a.log.Warnf("Retrying \"%s/%s.log\": %s", container, which, err) |
There was a problem hiding this comment.
This should not be a warning - it should match other logs.
|
Thanks, updated:
About testing with a real cluster: I don't think What I can do is replace the fake streamer with a small in-process HTTP/2 server that sends part of the log, then GOAWAY, then closes the connection, and run a real rest client against it. That covers the real client-go and net/http error path, including the error text we match, without a cluster. Would that work for you? |
6911027 to
8980d8c
Compare
We had only one instance of this issue with a real cluster. We don't know why it happened so reproducing it will be hard.
Using test server sounds like a better direction. We want to test that we retry after goaway error and and do not retry after other errors. |
| return | ||
| } | ||
|
|
||
| a.log.Debugf("Retrying \"%s/%s.log\": %s", container, which, err) |
| elapsed := time.Since(start).Seconds() | ||
| rate := float64(n) / float64(1024*1024) / elapsed | ||
| a.log.Debugf("Gathered \"%s/%s.log\" in %.3f seconds (%.2f MiB/s)", | ||
| container, which, elapsed, rate) |
There was a problem hiding this comment.
This log was confusing before this change after we failed to copy the log. Now that we retry 3 times it is worse.
Let change logging to this:
Gathering ... (1/3)
Failed ... in ... seconds: http2: server sent GOAWAY
Gathering ... (2/3)
Gathered ... in ... seconds (%.2f MiB/s)
One issue - we want to be able to tell how much time we spent on every log. Maybe the failure log should include also the time?
Example gather plot - the horizontal lines are log downloads:
The download start time is computed from log timestamp and the measured time.
With these logs we can show successful and failed logs attempts.
|
Updated:
|
8980d8c to
6f57701
Compare
|
@nirs this is ready for another look whenever you have time. CI is waiting for approval on the latest push. |
When the api server, or a load balancer in front of it, closes the connection gracefully during a log transfer, it sends an HTTP/2 GOAWAY frame, io.Copy fails, and we keep the partial log on disk with no indication that it is incomplete. The longest logs are the most likely to be hit, since they hold the connection open the longest. Retry the copy up to 3 times when the stream was interrupted by a GOAWAY. Every attempt opens a new stream and creates the log again, truncating the data copied by the previous attempt, since the logs API cannot resume an interrupted transfer. Every attempt is logged with its elapsed time, including failed ones, so we can tell how much time we spent on every log. The error is detected by matching the error text. errors.Is and errors.As cannot work here since net/http bundles a private copy of the http2 package, using an unexported error type (golang/go#28930). Fixes nirs#178 Signed-off-by: Haim Dimer <haim@dimer.org>
|
Rebased onto main to resolve the go.mod conflict from the dependency update; code unchanged. |
6f57701 to
6f82ee9
Compare
|
@hdimer Thanks for updating, I'll try to review again this week. |
Fixes #178.
When the api server sends a GOAWAY mid-transfer,
io.Copyfails and we keep whatever was received on disk, with no indication that the log is incomplete.gatherContainerLognow retries the copy up to 3 times on a GOAWAY. Every attempt opens a new stream and callsCreateContainerLogagain, which truncates the data copied by the previous attempt, since the logs API cannot resume a transfer.Detection is the string match you suggested in the issue:
errors.Is/errors.Ascannot work because net/http bundles a private copy of the http2 package with an unexported error type (golang/go#28930).To make the retry testable, the body of
gatherContainerLogmoved intocopyContainerLog, which takes a one-methodlogStreamerinterface that*rest.Requestalready satisfies. The newpkg/gather/logs_test.godrives it with a fake stream that fails after N bytes: retry on GOAWAY, no retry on other errors, the attempt limit, a failure to reopen the stream, and no log file created when the container has no stream.A few choices worth calling out, happy to change any of them:
k8s.io/client-go/util/retry.OnError. It adds backoff sleeps we do not need here, and the asymmetry (retry a failed copy, never retry the firstStream()failure, which is the expected PodInitializing race) needs plumbing around it anyway.string(which)conversion on a line the patch had to move anyway.Tested with
make unit-tests,golangci-lint run ./...andgolangci-lint fmt. I did not run the e2e suite (needs clusters), and the GOAWAY path is not reachable from it.Written with AI assistance (Claude). I reviewed and tested the change myself.