Reconciler errors starting vulnerability report Jobs

[Starttoaster]

What steps did you take and what happened:

The operator can't create its vulnerabilityreport Jobs suddenly anymore because the reconciler is erroring out in a loop.

What did you expect to happen:

To have VulnerabilityReports in my cluster

Anything else you would like to add:

I have hundreds of logs like this in my trivy-operator Pod:

{"level":"error","ts":"2024-08-13T21:05:19Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-6df95dd948","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-6df95dd948","reconcileID":"5e6ac00c-0b62-4204-beb9-c6c57ec9cfd3","error":"illegal base64 data at input byte 17200; illegal base64 data at input byte 6627","errorCauses":[{"error":"illegal base64 data at input byte 17200"},{"error":"illegal base64 data at input byte 6627"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}
{"level":"error","ts":"2024-08-13T21:05:20Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-86488678bc","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-86488678bc","reconcileID":"0a1f355d-ded5-42ab-a4ef-f8b4e59f02ca","error":"illegal base64 data at input byte 18865","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222"}

Environment:

• Trivy-Operator version (use trivy-operator version): 0.22.0
• Kubernetes version (use kubectl version): 1.29.1
• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc): Ubuntu 22.04

[Starttoaster]

Downgrading to chart version 0.23.2 and app version 0.21.2 fixed the vulnerability reports, but now I'm getting reconciler errors on Role and ClusterRole scanners.

[tranthang2404]

scanJob.compressLogs: "true" --> scanJob.compressLogs: "false"

maybe problem in encode and decode base64 logs

[Starttoaster]

I'm not sure that would be it since it seems like an issue with the operator making the scan jobs, rather than parsing output out of them. But I added that option to my helm values file and will let you know.

[Starttoaster]

Confusingly to me, that seems to have fixed it. I see new VulnerabilityReport resources generated after I upgraded back to latest trivy-operator using the latest helm chart release, with scanJob.compressLogs: "false". Thanks for suggesting a workaround! Seems like maybe a bug with log compression then though, so I'll keep the Issue open in case it's something that can be fixed.