Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Occasional re-registering of synced-probe-list nodes #546

Open
alexey-yarmosh opened this issue Sep 23, 2024 · 1 comment
Open

Occasional re-registering of synced-probe-list nodes #546

alexey-yarmosh opened this issue Sep 23, 2024 · 1 comment

Comments

@alexey-yarmosh
Copy link
Member

In New Relic there are many repeated logs like Removed node *. and Registered new node *.. That indicates that either the API server or Redis was too slow to update at the expected frequency.

Most likely that is the reason why some probes are offline on the dashboard from time to time:
Worker can't sync with a external node -> removes it's nodes from fetchProbes -> AdoptedProbes syncs missing probes as offline.

@MartinKolarik
Copy link
Member

99% of the occurrences were caused by wrong redis configuration (memory overload), and the errors went close to zero since we fixed that.

The remaining occasional cases are caused on the node.js side by event loop blocking, which is hard to track down. It got better with #560 and #561, and there might be some other places in request handling causing the rest of the spikes, but since it hardly causes any issues at this point, I'm assigning low priority.

@MartinKolarik MartinKolarik changed the title Constant re-registering of synced-probe-list nodes Occasional re-registering of synced-probe-list nodes Nov 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants