Async::Job vs SolidQueue

[adenta]

Hey Sam,

Figured it would be good for the community to have some discussion on when to use Async::Job and when to use SolidQueue.

Do you agree Async::Job is something you use in Paralell to SolidQueue?

SolidQueue: Persistant, handles server restarts, durable. Usefule for CPU bound background data processing
Async::Job: Fiber friendly, useful for LLM jobs but does not gracefully handle idemptentcy and server restarts

Let me know if I'm missing anything, love your work as always.

[MadBomber]

I converted a demo app over to async job when I upgraded to ruby 3.4.6 on my M2/max MacStudio running against PG and ran into major crashes that were specific to that environment/hardware combination. I'm liking what I see in async job.

[ioquatix]

but does not gracefully handle idemptentcy and server restarts

The idempotency and failure behaviour depends completely on the queue implementation.

• In-process queue is fast but likely to loose data in the event of a crash.
• The redis queue is pretty robust and shouldn't loose data in normal operation or failure scenarios.

I encourage you to try running a job client and server, and the experiment with killing the job server, then restart it, to see how it works. Enable CONSOLE_LEVEL=debug logging for all the output.

[ioquatix]

Do you agree Async::Job is something you use in parallel to SolidQueue?

They definitely can be used in the same application. I think which option you choose would depend on your use case. Async::Job is better at I/O multiplexing during job execution so should have increased efficiency in many scenarios where background jobs are used (e.g. talking to slow APIs).

In addition, I would like to introduce a database backed queue in the future, so that we can have strong transactional safety for jobs that require it. In this case, I'll probably focus on blending the job queue with existing database records, since in the vast majority of cases that's my use case.

In opposition to that, apart from convenience, I don't think it's necessarily a good idea to use a database backed job queue if you don't need transactional safety – or consistency – e.g. consider if you need to roll back a database separately from your job server data store (whatever that may be). You should consider those traits and trade-offs depending on your requirements.