-
Notifications
You must be signed in to change notification settings - Fork 487
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(loki/src/k8s_events): correctly update health state for the component during operation #6385
base: main
Are you sure you want to change the base?
fix(loki/src/k8s_events): correctly update health state for the component during operation #6385
Conversation
85b7a36
to
7a73fc9
Compare
@@ -152,6 +156,7 @@ func (c *Component) Run(ctx context.Context) error { | |||
c.tasksMut.RUnlock() | |||
|
|||
if err := c.runner.ApplyTasks(ctx, tasks); err != nil { | |||
c.setHealth(err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this solves the issue in #6181 entirely.
ApplyTasks will return an error if the runner is closed, or if the context is already cancelled/exceeded via ctx.Err; we don't have a way to propagate errors directly from the underlying tasks as the worker interface is Run(ctx context.Context)
.
To make this happen we'd need to have a goroutine-safe mechanism of bubbling up the errors and sieving through them.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok so my approach is to change the worker interface as there aren't many implementations yet.
The ApplyTask
function will now collect returned errors and store within the caller's struct.
I add a new method for Worker interface to get the bubbled-up errors, GetWorkerErrors
to help setting the component's health properly.
PTAL if you have any concern re: goroutine-safe aspect. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've changed my approach and now choose one with smaller blast radius. That is, the eventController
retain any error when executing Run
method and there's a recurring actor in the component to set health based on the returned results.
Not exactly ideal since the component's health is not reflected right away but I think it's a worthwhile tradeoff
…nent during operation Signed-off-by: hainenber <[email protected]>
Signed-off-by: hainenber <[email protected]>
Signed-off-by: hainenber <[email protected]>
…ccordingly Signed-off-by: hainenber <[email protected]>
…tor to set health based on applied task result Signed-off-by: hainenber <[email protected]>
c184e12
to
61693df
Compare
Signed-off-by: hainenber <[email protected]>
This PR has not had any activity in the past 30 days, so the |
This PR has not had any activity in the past 30 days, so the |
PR Description
Which issue(s) this PR fixes
Fixes grafana/alloy#274
Notes to the Reviewer
PR Checklist