[Bug] Goose worker threads keep spinning when Redis connection is not available

[joodie]

No back-off or bail-out in worker if Redis consumer fails to connect.

When starting a goose worker with Redis broker, if the redis broker is not available (for instance, if the Redis connection uri is incorrect), the worker threads start up as normal and then keep spinning when trying to fetch the next job.

There is no backoff or halt in this "fetch loop" so default behaviour is that 4 threads are continuously trying to connect to redis and fetch the next job with no visibility that anything is wrong from from the "outside" system code. This results in significant system load and lots of exceptions filling the logs.

Versions

1. Goose: 0.4.0
2. Clojure: 1.11.0

Cause

In goose.brokers.redis.consumer/run the job fetching from redis and the call to execute the fetch job is wrapped in a log-on-exceptions macro that catches, logs, and then ignores, any exception.

Expected behaviours

• Ideally, redis worker should fail to start if the Redis connection cannot be established (possibly after some retries).
• If a redis connection fails (i.e. Redis server is shut down or otherwise suddenly unavailable) worker probably should retry with backoff before failing completely. Worker should not keep spinning, "spamming" the server and logs.
• Worker health status should be inspectable / there should be some way to detect a "total system failure" and shut down the worker: this probably requires some interaction with/callback to the system that started the worker.