Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

backupccl: v23.1.13: panic in authhandler #117227

Closed
cockroach-sentry opened this issue Jan 2, 2024 · 5 comments
Closed

backupccl: v23.1.13: panic in authhandler #117227

cockroach-sentry opened this issue Jan 2, 2024 · 5 comments
Labels
A-disaster-recovery C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. no-issue-activity O-sentry Originated from an in-the-wild panic report. P-3 Issues/test failures with no fix SLA T-disaster-recovery X-stale

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Jan 2, 2024 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4791472886/?referrer=webhooks_plugin

Panic Message:

panic.go:884: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | runtime.panicmem
  | 	GOROOT/src/runtime/panic.go:260
  | runtime.sigpanic
  | 	GOROOT/src/runtime/signal_unix.go:839
  | golang.org/x/oauth2/authhandler.authHandlerSource.Token
  | 	golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go:80
  | golang.org/x/oauth2.(*reuseTokenSource).Token
  | 	golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go:304
  | golang.org/x/oauth2.(*Transport).RoundTrip
  | 	golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go:45
  | net/http.send
  | 	GOROOT/src/net/http/client.go:251
  | net/http.(*Client).send
  | 	GOROOT/src/net/http/client.go:175
  | net/http.(*Client).do
  | 	GOROOT/src/net/http/client.go:715
  | net/http.(*Client).Do
  | 	GOROOT/src/net/http/client.go:581
  | cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go:834
  | cloud.google.com/go/storage.run.func1
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go:67
  | cloud.google.com/go/internal.retry
  | 	cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:40
  | cloud.google.com/go/internal.Retry
  | 	cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go:33
  | cloud.google.com/go/storage.run
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go:65
  | cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go:833
  | cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go:887
  | cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader
  | 	cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go:118
  | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFileAt.func1
  | 	github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:291
  | github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open.func1
  | 	github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:227
  | github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry.func1
  | 	github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:110
  | github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:195
  | github.com/cockroachdb/cockroach/pkg/util/retry.Options.Do
  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:172
  | github.com/cockroachdb/cockroach/pkg/util/retry.WithMaxAttempts
  | 	github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:194
  | github.com/cockroachdb/cockroach/pkg/cloud.DelayedRetry
  | 	github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:109
  | github.com/cockroachdb/cockroach/pkg/cloud.(*ResumingReader).Open
  | 	github.com/cockroachdb/cockroach/pkg/cloud/cloud_io.go:224
  | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFileAt
  | 	github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:300
  | github.com/cockroachdb/cockroach/pkg/cloud/gcp.(*gcsStorage).ReadFile
  | 	github.com/cockroachdb/cockroach/pkg/cloud/gcp/gcs_storage.go:276
  | github.com/cockroachdb/cockroach/pkg/cloud.(*esWrapper).ReadFile
  | 	github.com/cockroachdb/cockroach/pkg/cloud/impl_registry.go:331
  | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest.containsManifest
  | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go:71
  | github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest.ResolveDest
  | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go:169
  | github.com/cockroachdb/cockroach/pkg/ccl/backupccl.(*backupResumer).Resume
  | 	github.com/cockroachdb/cockroach/pkg/ccl/backupccl/backup_job.go:503
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:884: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets):

cockroach/pkg/ccl/backupccl/backup_job.go

Lines 502 to 504 in 8d065df

var err error
backupDest, err = backupdest.ResolveDest(ctx, p.User(), details.Destination, details.EndTime,
details.IncrementalFrom, p.ExecCfg())

cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go

Lines 168 to 170 in 8d065df

defer defaultStore.Close()
exists, err := containsManifest(ctx, defaultStore)
if err != nil {

cockroach/pkg/ccl/backupccl/backupdest/backup_destination.go

Lines 70 to 72 in 8d065df

func containsManifest(ctx context.Context, exportStore cloud.ExternalStorage) (bool, error) {
r, err := exportStore.ReadFile(ctx, backupbase.BackupManifestName)
if err != nil {

cockroach/pkg/cloud/impl_registry.go

Lines 330 to 332 in 8d065df

func (e *esWrapper) ReadFile(ctx context.Context, basename string) (ioctx.ReadCloserCtx, error) {
r, err := e.ExternalStorage.ReadFile(ctx, basename)
if err != nil {

cockroach/pkg/cloud/gcp/gcs_storage.go

Lines 275 to 277 in 8d065df

func (g *gcsStorage) ReadFile(ctx context.Context, basename string) (ioctx.ReadCloserCtx, error) {
reader, _, err := g.ReadFileAt(ctx, basename, 0)
return reader, err

cockroach/pkg/cloud/gcp/gcs_storage.go

Lines 299 to 301 in 8d065df

if err := r.Open(ctx); err != nil {
if errors.Is(err, gcs.ErrObjectNotExist) {

cockroach/pkg/cloud/cloud_io.go

Lines 223 to 225 in 8d065df

func (r *ResumingReader) Open(ctx context.Context) error {
return DelayedRetry(ctx, "Open", r.ErrFn, func() error {
var readErr error

cockroach/pkg/cloud/cloud_io.go

Lines 108 to 110 in 8d065df

return retry.WithMaxAttempts(ctx, base.DefaultRetryOptions(), MaxDelayedRetryAttempts, func() error {
err := fn()

cockroach/pkg/util/retry/retry.go

Lines 193 to 195 in 8d065df

opts.MaxRetries = n - 1 // >= 1
return opts.Do(ctx, func(ctx context.Context) error {
return fn()

cockroach/pkg/util/retry/retry.go

Lines 171 to 173 in 8d065df

for r := StartWithCtx(ctx, opts); r.Next(); {
err = fn(ctx)
if err == nil {

cockroach/pkg/util/retry/retry.go

Lines 194 to 196 in 8d065df

return opts.Do(ctx, func(ctx context.Context) error {
return fn()
})

cockroach/pkg/cloud/cloud_io.go

Lines 109 to 111 in 8d065df

return retry.WithMaxAttempts(ctx, base.DefaultRetryOptions(), MaxDelayedRetryAttempts, func() error {
err := fn()
if err == nil {

cockroach/pkg/cloud/cloud_io.go

Lines 226 to 228 in 8d065df

r.Reader, readErr = r.Opener(ctx, r.Pos)
if readErr != nil {

cockroach/pkg/cloud/gcp/gcs_storage.go

Lines 290 to 292 in 8d065df

func(ctx context.Context, pos int64) (io.ReadCloser, error) {
return g.bucket.Object(object).NewRangeReader(ctx, pos, -1)
}, // opener

cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go#L117-L119
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L886-L888
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L832-L834
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go#L64-L66
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go#L32-L34
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go#L39-L41
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go#L66-L68
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go#L833-L835
GOROOT/src/net/http/client.go#L580-L582
GOROOT/src/net/http/client.go#L714-L716
GOROOT/src/net/http/client.go#L174-L176
GOROOT/src/net/http/client.go#L250-L252
golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go#L44-L46
golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go#L303-L305
golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go#L79-L81
GOROOT/src/runtime/signal_unix.go#L838-L840
GOROOT/src/runtime/panic.go#L259-L261
GOROOT/src/runtime/panic.go#L883-L885

pkg/ccl/backupccl/backup_job.go in pkg/ccl/backupccl.(*backupResumer).Resume at line 503
pkg/ccl/backupccl/backupdest/backup_destination.go in pkg/ccl/backupccl/backupdest.ResolveDest at line 169
pkg/ccl/backupccl/backupdest/backup_destination.go in pkg/ccl/backupccl/backupdest.containsManifest at line 71
pkg/cloud/impl_registry.go in pkg/cloud.(*esWrapper).ReadFile at line 331
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFile at line 276
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFileAt at line 300
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open at line 224
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry at line 109
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts at line 194
pkg/util/retry/retry.go in pkg/util/retry.Options.Do at line 172
pkg/util/retry/retry.go in pkg/util/retry.WithMaxAttempts.func1 at line 195
pkg/cloud/cloud_io.go in pkg/cloud.DelayedRetry.func1 at line 110
pkg/cloud/cloud_io.go in pkg/cloud.(*ResumingReader).Open.func1 at line 227
pkg/cloud/gcp/gcs_storage.go in pkg/cloud/gcp.(*gcsStorage).ReadFileAt.func1 at line 291
cloud.google.com/go/storage/external/com_google_cloud_go_storage/reader.go in cloud.google.com/go/storage.(*ObjectHandle).NewRangeReader at line 118
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader at line 887
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2 at line 833
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run at line 65
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.Retry at line 33
cloud.google.com/go/internal/external/com_google_cloud_go/internal/retry.go in cloud.google.com/go/internal.retry at line 40
cloud.google.com/go/storage/external/com_google_cloud_go_storage/invoke.go in cloud.google.com/go/storage.run.func1 at line 67
cloud.google.com/go/storage/external/com_google_cloud_go_storage/http_client.go in cloud.google.com/go/storage.(*httpStorageClient).NewRangeReader.func2.1 at line 834
GOROOT/src/net/http/client.go in net/http.(*Client).Do at line 581
GOROOT/src/net/http/client.go in net/http.(*Client).do at line 715
GOROOT/src/net/http/client.go in net/http.(*Client).send at line 175
GOROOT/src/net/http/client.go in net/http.send at line 251
golang.org/x/oauth2/external/org_golang_x_oauth2/transport.go in golang.org/x/oauth2.(*Transport).RoundTrip at line 45
golang.org/x/oauth2/external/org_golang_x_oauth2/oauth2.go in golang.org/x/oauth2.(*reuseTokenSource).Token at line 304
golang.org/x/oauth2/authhandler/external/org_golang_x_oauth2/authhandler/authhandler.go in golang.org/x/oauth2/authhandler.authHandlerSource.Token at line 80
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 839
GOROOT/src/runtime/panic.go in runtime.panicmem at line 260
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884

Tags

Tag Value
Command mt start-sql
Environment v23.1.13
Go Version go1.19.13
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.13
Cockroach SHA 8d065df
# of CPUs 16
# of Goroutines 256

Jira issue: CRDB-35073
@cockroach-sentry cockroach-sentry added O-sentry Originated from an in-the-wild panic report. C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. labels Jan 2, 2024
@yuzefovich yuzefovich changed the title Sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | runtime.p... backupccl: v23.1.13: panic in authhandler Jan 23, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Jan 23, 2024

cc @cockroachdb/disaster-recovery

@yuzefovich
Copy link
Member

This panic happened a few times on CC https://cockroachlabsgcp.splunkcloud.com/en-US/app/search/search?q=search%20containsManifest%20%22a%20panic%20has%20occurred%22&display.page.search.mode=smart&dispatch.sample_ratio=1&workload_pool=standard_perf&earliest=-30d%40d&latest=now&sid=1706036498.1083546.

@dt
Copy link
Member

dt commented Jan 25, 2024 •
edited
Loading

Looks like golang/oauth2#583

Sounds like it could be an incorrect/malformed credentials file being used.

@dt dt added the P-3 Issues/test failures with no fix SLA label Jan 25, 2024
@yuzefovich yuzefovich mentioned this issue Mar 18, 2024
Closed
@github-actions GitHub Actions
Copy link

We have marked this issue as stale because it has been inactive for
12 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Feb 10, 2025
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Feb 10, 2025

Please add branch-* labels to identify which branch(es) this C-bug affects.

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-disaster-recovery C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. no-issue-activity O-sentry Originated from an in-the-wild panic report. P-3 Issues/test failures with no fix SLA T-disaster-recovery X-stale
Projects
No open projects
Disaster Recovery Backlog
Status: Backlog
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dt @yuzefovich @cockroach-sentry