BigQuery output plugin error: invalid memory address or nil pointer dereference

[saraghds]

Relevant telegraf.conf

[[inputs.internal]]
  [inputs.internal.tags]
    user = "user"
    workspace = "workspace"
[[outputs.bigquery]]
  alias = "monitoring"
  credentials_file = "/path/to/oauth_creds.json"
  project = "project-name"
  dataset = "data-set-name"
  replace_hyphen_to = "_"
  namepass = ["internal_write*"]

Logs from Telegraf

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x132be2d]

goroutine 198 [running]:
golang.org/x/oauth2/authhandler.authHandlerSource.Token({{0x7846c60, 0xc000dc7c20}, 0xc0001feaf0, 0x0, {0x0, 0x0}, 0x0})
	/home/saraghodsi/go/pkg/mod/golang.org/x/[email protected]/authhandler/authhandler.go:80 +0x1ad
golang.org/x/oauth2.(*reuseTokenSource).Token(0xc000dc7ef0)
	/home/saraghodsi/go/pkg/mod/golang.org/x/[email protected]/oauth2.go:307 +0xd5
golang.org/x/oauth2.(*Transport).RoundTrip(0xc000affd80, 0xc004184b00)
	/home/saraghodsi/go/pkg/mod/golang.org/x/[email protected]/transport.go:45 +0xe2
net/http.send(0xc004184b00, {0x77ea4e0, 0xc000affd80}, {0x8?, 0x6a2baa0?, 0x0?})
	/opt/golang/1.20.4/src/net/http/client.go:252 +0x5f7
net/http.(*Client).send(0xc000dc7f80, 0xc004184b00, {0xc0040c16d0?, 0xc003078d10?, 0x0?})
	/opt/golang/1.20.4/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc000dc7f80, 0xc004184b00)
	/opt/golang/1.20.4/src/net/http/client.go:716 +0x8fb
net/http.(*Client).Do(...)
	/opt/golang/1.20.4/src/net/http/client.go:582
google.golang.org/api/internal/gensupport.send({0x7846c98?, 0xc0040341b0}, 0x6c16a7a?, 0xc004184a00)
	/home/saraghodsi/go/pkg/mod/google.golang.org/[email protected]/internal/gensupport/send.go:61 +0xf2
google.golang.org/api/internal/gensupport.SendRequest({0x7846c98, 0xc0040341b0}, 0x6bebf39?, 0xc004184a00)
	/home/saraghodsi/go/pkg/mod/google.golang.org/[email protected]/internal/gensupport/send.go:54 +0x6d
google.golang.org/api/bigquery/v2.(*TabledataInsertAllCall).doRequest(0xc0040c1d70, {0x6be0a5a, 0x4})
	/home/saraghodsi/go/pkg/mod/google.golang.org/[email protected]/bigquery/v2/bigquery-gen.go:14544 +0x805
google.golang.org/api/bigquery/v2.(*TabledataInsertAllCall).Do(0xc002ea8d70, {0x0?, 0x6c96b0d?, 0x7fc159656408?})
	/home/saraghodsi/go/pkg/mod/google.golang.org/[email protected]/bigquery/v2/bigquery-gen.go:14556 +0x6d
cloud.google.com/go/bigquery.(*Inserter).putMulti.func1()
	/home/saraghodsi/go/pkg/mod/cloud.google.com/go/[email protected]/inserter.go:185 +0x7f
cloud.google.com/go/bigquery.runWithRetryExplicit.func1()
	/home/saraghodsi/go/pkg/mod/cloud.google.com/go/[email protected]/bigquery.go:222 +0x38
cloud.google.com/go/internal.retry({0x7846c98, 0xc0040341b0}, {0x3b9aca00, 0x773594000, 0x4000000000000000, 0x0}, 0xc0040c1c70, 0x7020c50)
	/home/saraghodsi/go/pkg/mod/cloud.google.com/[email protected]/internal/retry.go:40 +0x7a
cloud.google.com/go/internal.Retry(...)
	/home/saraghodsi/go/pkg/mod/cloud.google.com/[email protected]/internal/retry.go:33
cloud.google.com/go/bigquery.runWithRetryExplicit({0x7846c98?, 0xc0040341b0?}, 0xc004186300?, {0xb0b4280?, 0x0?, 0x5?})
	/home/saraghodsi/go/pkg/mod/cloud.google.com/go/[email protected]/bigquery.go:221 +0x89
cloud.google.com/go/bigquery.runWithRetry(...)
	/home/saraghodsi/go/pkg/mod/cloud.google.com/go/[email protected]/bigquery.go:211
cloud.google.com/go/bigquery.(*Inserter).putMulti(0xc002ea8ee8, {0x7846c98, 0xc004034420}, {0xc003fa1500?, 0x5d74aa0?, 0x1c002ea8e68?})
	/home/saraghodsi/go/pkg/mod/cloud.google.com/go/[email protected]/inserter.go:183 +0x30f
cloud.google.com/go/bigquery.(*Inserter).Put(0xc003d50780?, {0x7846c60?, 0xc004034150?}, {0x5b2b300, 0xc000626000})
	/home/saraghodsi/go/pkg/mod/cloud.google.com/go/[email protected]/inserter.go:103 +0x13e
github.com/influxdata/telegraf/plugins/outputs/bigquery.(*BigQuery).insertToTable(0xc0001ca620, {0xc0020d6330, 0xe}, {0xc003d50780, 0x5, 0x8})
	/home/saraghodsi/telegraf/plugins/outputs/bigquery/bigquery.go:197 +0x23c
github.com/influxdata/telegraf/plugins/outputs/bigquery.(*BigQuery).Write.func1({0xc0020d6330?, 0x0?}, {0xc003d50780?, 0xc0029ba210?, 0x0?})
	/home/saraghodsi/telegraf/plugins/outputs/bigquery/bigquery.go:100 +0x78
created by github.com/influxdata/telegraf/plugins/outputs/bigquery.(*BigQuery).Write
	/home/saraghodsi/telegraf/plugins/outputs/bigquery/bigquery.go:98 +0x90

System info

Telegraf version 1.27.0-ddecf6f7, Ubuntu 20.04.2 LTS

Docker

No response

Steps to reproduce

1. Create .conf file with outputs.bigquery correct credentials file and the right project and dataset name.
2. Start telegraf using following script:

start-stop-daemon --start \
  --background \
  --no-close \
  --pidfile $HOME/telegraf.pid \
  --exec $HOME/telegraf/telegraf \
  -- --config $HOME/telegraf.conf \
	  --pidfile $HOME/telegraf.pid

Expected behavior

Data from inputs.internal plugin being written to bigquery.

Actual behavior

Immediately the error is shown in the logs. It's important to note that other output plugins like influxdb_v2 have worked fine.

Additional info

No response

[powersj]

Hi,

This looks like golang/oauth2#583 as the authhandler/authhandler.go:80 refers to:

code, state, err := source.authHandler(url)

Questions:

1. Has this ever worked?
2. If you use the gcloud cli to login does it work then?

[saraghds]

It looks like my .json file was not the correct one. Although I still get an error when using the correct one:

2023-05-05T17:45:07Z E! [outputs.bigquery::monitoring] inserting metric "internal_write" failed: Post "https://bigquery.googleapis.com/bigquery/v2/projects/project-name/datasets/data-set-name/tables/internal_write/insertAll?alt=json&prettyPrint=false": Post "https://oauth2.googleapis.com/token": context canceled
2023-05-05T17:45:17Z E! [outputs.bigquery::monitoring] inserting metric "internal_write" failed: Post "https://bigquery.googleapis.com/bigquery/v2/projects/project-name/datasets/data-set-name/tables/internal_write/insertAll?alt=json&prettyPrint=false": Post "https://oauth2.googleapis.com/token": context canceled
2023-05-05T17:45:27Z E! [outputs.bigquery::monitoring] inserting metric "internal_write" failed: Post "https://bigquery.googleapis.com/bigquery/v2/projects/project-name/datasets/data-set-name/tables/internal_write/insertAll?alt=json&prettyPrint=false": Post "https://oauth2.googleapis.com/token": context canceled
2023-05-05T17:45:37Z E! [outputs.bigquery::monitoring] inserting metric "internal_write" failed: Post "https://bigquery.googleapis.com/bigquery/v2/projects/project-name/datasets/data-set-name/tables/internal_write/insertAll?alt=json&prettyPrint=false": Post "https://oauth2.googleapis.com/token": context canceled

[saraghds]

I have created the table fully with the schema. But not sure, maybe I did it incorrectly?

[powersj]

It looks like my .json file was not the correct one.

Good to know!

context canceled

Can you bump up your timeout? Say timeout = "30s" for now?

How many metrics are you pushing up at a time?

[saraghds]

I still get the same error when using the timeout setting. I'm pushing the internal_write metrics (7 of them) every 10 seconds.

[powersj]

I still get the same error when using the timeout setting. I'm pushing the internal_write metrics (7 of them) every 10 seconds.

So it does not sound like you are sending many metrics at all :)

Can you share more logs? Is it the same message?
Just out of curiosity, if you increase the flush_interval to say 60 seconds does that also help?
Are you aware of any limits on your account?

[saraghds]

Yes, it's the same message:

2023-05-19T15:48:37Z E! [outputs.bigquery::monitoring] inserting metric "internal_write" failed: Post "https://bigquery.googleapis.com/bigquery/v2/projects/project-name/datasets/data-set-name/tables/internal_write/insertAll?alt=json&prettyPrint=false": Post "https://oauth2.googleapis.com/token": context canceled

I increased the flush interval and timeout to 60s, still got the message.
I don't believe I have any limits on my account, but how do I check that?

[powersj]

I don't believe I have any limits on my account, but how do I check that?

Not actually certain :) the error is not really something to do with telegraf itself, but rather a message about trying to get your token and timing out. It makes me wonder if your token config is wrong, your system cannot get to the token service itself, or some other thing is stopping you from getting access to the authorization.

[telegraf-tiger]

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!