Skip to content

logs: Retry log stream interrupted by GOAWAY - #187

Open
hdimer wants to merge 1 commit into
nirs:mainfrom
hdimer:logs-retry-goaway
Open

hdimer wants to merge 1 commit into
nirs:mainfrom
hdimer:logs-retry-goaway

Conversation

@hdimer

@hdimer hdimer commented Aug 18, 2026

Copy link
Copy Markdown

Fixes #178.

When the api server sends a GOAWAY mid-transfer, io.Copy fails and we keep whatever was received on disk, with no indication that the log is incomplete. gatherContainerLog now retries the copy up to 3 times on a GOAWAY. Every attempt opens a new stream and calls CreateContainerLog again, which truncates the data copied by the previous attempt, since the logs API cannot resume a transfer.

Detection is the string match you suggested in the issue: errors.Is/errors.As cannot work because net/http bundles a private copy of the http2 package with an unexported error type (golang/go#28930).

To make the retry testable, the body of gatherContainerLog moved into copyContainerLog, which takes a one-method logStreamer interface that *rest.Request already satisfies. The new pkg/gather/logs_test.go drives it with a fake stream that fails after N bytes: retry on GOAWAY, no retry on other errors, the attempt limit, a failure to reopen the stream, and no log file created when the container has no stream.

A few choices worth calling out, happy to change any of them:

  • Not using k8s.io/client-go/util/retry.OnError. It adds backoff sleeps we do not need here, and the asymmetry (retry a failed copy, never retry the first Stream() failure, which is the expected PodInitializing race) needs plumbing around it anyway.
  • Failing to reopen the stream on a retry is now a warning rather than a debug message, otherwise the user sees "Retrying ..." followed by silence and a truncated log.
  • No truncation marker for the case where all 3 attempts fail (problem 1 in the issue). That changes the output layout, so I left it out. Glad to add it here or in a follow-up if you want it.
  • Dropped a no-op string(which) conversion on a line the patch had to move anyway.

Tested with make unit-tests, golangci-lint run ./... and golangci-lint fmt. I did not run the e2e suite (needs clusters), and the GOAWAY path is not reachable from it.

Written with AI assistance (Claude). I reviewed and tested the change myself.

@hdimer
hdimer marked this pull request as ready for review August 19, 2026 00:06
@nirs
nirs force-pushed the logs-retry-goaway branch from e0dd905 to 6911027 Compare October 13, 2026 17:05
Comment thread pkg/gather/logs.go Outdated
n, err = io.Copy(dst, src)

src.Close()
dst.Close()

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Closing manually is error prone.

We should extract a function doing streaming and copying and use defer in the function. Then the retry loop can be much simpler.

Closing dst without checking for errors hides failure to flush data to the log. We should check the error and warn in this case.

Comment thread pkg/gather/logs.go Outdated
// failing to stream it again is not expected.
a.log.Warnf("Cannot get log for \"%s/%s\": %s", container, which, err)
return
}

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need special handling based on number of attempts? If we treat Stream error as fatal error, it does not matter if this is the first or second attempt.

The only reason may be if we log a different message about partial log fetched by the previous attempt.

Comment thread pkg/gather/logs_test.go

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests use too much mocking - they may succeed when real log streaming will fail and increase maintenance cost. We should test this using real cluster if we can trigger the goway error.

Comment thread pkg/gather/logs.go Outdated
}

if attempt < logStreamAttempts && isGoAwayError(err) {
a.log.Warnf("Retrying \"%s/%s.log\": %s", container, which, err)

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not be a warning - it should match other logs.

@hdimer

hdimer commented Oct 13, 2026

Copy link
Copy Markdown
Author

Thanks, updated:

  • Extracted streamContainerLog, which is mostly the original function body with deferred closes. A dst.Close() failure is now logged as a warning. The retry loop only decides if we should retry.
  • A Stream() error is handled the same way in every attempt, as a debug message like before.
  • "Retrying" is now a debug message.

About testing with a real cluster: I don't think --goaway-chance can trigger this. The apiserver goaway filter only sets Connection: close, so the Go http2 server sends a graceful GOAWAY and waits for open streams to complete before closing the connection. An in-flight log stream finishes normally. The error in #178 needs the connection closed while the stream is still open, like a load balancer or an apiserver going down during the transfer. With a single apiserver on minikube the retry would fail too, since the server is down.

What I can do is replace the fake streamer with a small in-process HTTP/2 server that sends part of the log, then GOAWAY, then closes the connection, and run a real rest client against it. That covers the real client-go and net/http error path, including the error text we match, without a cluster. Would that work for you?

@hdimer
hdimer force-pushed the logs-retry-goaway branch from 6911027 to 8980d8c Compare October 13, 2026 18:44
@nirs

nirs commented Oct 13, 2026

Copy link
Copy Markdown
Owner

About testing with a real cluster: I don't think --goaway-chance can trigger this. The apiserver goaway filter only sets Connection: close, so the Go http2 server sends a graceful GOAWAY and waits for open streams to complete before closing the connection. An in-flight log stream finishes normally. The error in #178 needs the connection closed while the stream is still open, like a load balancer or an apiserver going down during the transfer. With a single apiserver on minikube the retry would fail too, since the server is down.

We had only one instance of this issue with a real cluster. We don't know why it happened so reproducing it will be hard.

What I can do is replace the fake streamer with a small in-process HTTP/2 server that sends part of the log, then GOAWAY, then closes the connection, and run a real rest client against it. That covers the real client-go and net/http error path, including the error text we match, without a cluster. Would that work for you?

Using test server sounds like a better direction. We want to test that we retry after goaway error and and do not retry after other errors.

Comment thread pkg/gather/logs.go Outdated
return
}

a.log.Debugf("Retrying \"%s/%s.log\": %s", container, which, err)

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add "(attempt %d/total)"?

Comment thread pkg/gather/logs.go
elapsed := time.Since(start).Seconds()
rate := float64(n) / float64(1024*1024) / elapsed
a.log.Debugf("Gathered \"%s/%s.log\" in %.3f seconds (%.2f MiB/s)",
container, which, elapsed, rate)

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This log was confusing before this change after we failed to copy the log. Now that we retry 3 times it is worse.

Let change logging to this:

Gathering ... (1/3)
Failed ... in ... seconds: http2: server sent GOAWAY
Gathering ... (2/3)
Gathered ... in ... seconds (%.2f MiB/s)

One issue - we want to be able to tell how much time we spent on every log. Maybe the failure log should include also the time?

Example gather plot - the horizontal lines are log downloads:

Image

The download start time is computed from log timestamp and the measured time.

With these logs we can show successful and failed logs attempts.

@hdimer

hdimer commented Oct 16, 2026

Copy link
Copy Markdown
Author

Updated:

  • The tests now run a real client-go clientset against a small HTTP/2 server that writes the frames directly: a complete response, partial data then GOAWAY and close, or partial data then close without GOAWAY. They check that we retry after GOAWAY, up to 3 attempts, and do not retry after other errors. The GOAWAY case fails with the same error as in Retry log streaming on HTTP/2 GOAWAY #178. With a real client the logStreamer interface was not needed anymore, so the retry loop is back in gatherContainerLog.
  • Logging is per attempt now, with the time in the failure line:
    Gathering "ns/pod/container/current.log" (1/3)
    Failed "ns/pod/container/current.log" in 0.002 seconds: http2: server sent GOAWAY and closed the connection; ...
    Gathering "ns/pod/container/current.log" (2/3)
    Gathered "ns/pod/container/current.log" in 0.001 seconds (0.00 MiB/s)
    
    The separate "Retrying" message is gone, since (2/3) already says it.

@hdimer
hdimer force-pushed the logs-retry-goaway branch from 8980d8c to 6f57701 Compare October 16, 2026 00:27
@hdimer

hdimer commented Nov 3, 2026

Copy link
Copy Markdown
Author

@nirs this is ready for another look whenever you have time. CI is waiting for approval on the latest push.

When the api server, or a load balancer in front of it, closes the
connection gracefully during a log transfer, it sends an HTTP/2 GOAWAY
frame, io.Copy fails, and we keep the partial log on disk with no
indication that it is incomplete. The longest logs are the most likely
to be hit, since they hold the connection open the longest.

Retry the copy up to 3 times when the stream was interrupted by a
GOAWAY. Every attempt opens a new stream and creates the log again,
truncating the data copied by the previous attempt, since the logs API
cannot resume an interrupted transfer.

Every attempt is logged with its elapsed time, including failed ones,
so we can tell how much time we spent on every log.

The error is detected by matching the error text. errors.Is and
errors.As cannot work here since net/http bundles a private copy of the
http2 package, using an unexported error type (golang/go#28930).

Fixes nirs#178

Signed-off-by: Haim Dimer <haim@dimer.org>
@hdimer

hdimer commented Nov 9, 2026

Copy link
Copy Markdown
Author

Rebased onto main to resolve the go.mod conflict from the dependency update; code unchanged.

@hdimer
hdimer force-pushed the logs-retry-goaway branch from 6f57701 to 6f82ee9 Compare November 9, 2026 18:46
@nirs

nirs commented Nov 9, 2026

Copy link
Copy Markdown
Owner

@hdimer Thanks for updating, I'll try to review again this week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Retry log streaming on HTTP/2 GOAWAY

2 participants