[18.0][ADD] queue_job: per DB channel managers with UI configuration and hot reload - #962
Open
guewen wants to merge 2 commits into
Open
[18.0][ADD] queue_job: per DB channel managers with UI configuration and hot reload#962guewen wants to merge 2 commits into
guewen wants to merge 2 commits into
Conversation
Contributor
|
Hi @sbidoul, |
guewen
force-pushed
the
18.0-queue-job-channel-config
branch
4 times, most recently
from
August 18, 2026 12:02
9b9e3fd to
934ccb4
Compare
guewen
changed the title
Member
Author
|
Hi @sbidoul , can I have your thoughts on this, before I start completing the tests and docs? |
Contributor
|
Hi @sbidoul, |
sbidoul
reviewed
Aug 28, 2026
sbidoul
left a comment
sbidoul
left a comment
Member
There was a problem hiding this comment.
Hi Guewen,
I have not had time to do a proper review, but I think I'm on board with this.
Great solution!
I have some doubts about the round robin rule, but on the other hand I don't have serious multi-db use cases myself so this can be addressed later.
| capacity: int = 0 | ||
| sequential: bool = False | ||
| throttle: int = 0 | ||
| paused: bool = False |
Member
There was a problem hiding this comment.
Handle the new default subchannel capacity too?
Member
There was a problem hiding this comment.
It looks like the subchannel pull request (#767) has a merge conflict. I'll take a look at it today.
Member
There was a problem hiding this comment.
Ah I thought that was merged already.
Member
Author
There was a problem hiding this comment.
I rebased this branch with the default subchannel capacity now that is is merged
Member
Author
|
I'm writing a benchmark script to have some comparison points (I'll open another PR for the script at some point). It also demonstrates an interesting effect of the round-robin (nothing surprising but nice to see it with numbers). In the current state of my benchmark, the script enqueues 5k jobs in db0, then 5k in db1, then 5k in db2. With the global channel manager, the jobrunner executes the jobs strictly by creation order, so the 5k jobs of db0 are executed, then those of db1 and finally those of db2. The round-robin (yet to test in real situation, which I do not have as we have a single DB in production), might help in situations where a large DB clutters the queue with a large quantity of jobs and starve other databases. Do not give much value to the number otherwise, it is only a single run of each. global channel manager The throughput per database is much higher than the round-robin version: this is because the results are based on when the 5000 jobs are started and when they are all done, and all jobs the bench0 are done before the others, the 5k jobs are done much earlier. round-robin In the round-robin version, the throughput is equal for each database, because their jobs are all starting and ending at about the same points. memo for myself on the benchmark |
Member
A single database running multiple companies can also find itself at the mercy of strict creation order. |
Member
Author
This one won't be solved though |
guewen
force-pushed
the
18.0-queue-job-channel-config
branch
3 times, most recently
from
September 11, 2026 14:08
9578eb4 to
b037b8f
Compare
Not used at this point by the jobrunner, but the changes on channels trigger a notify to the jobrunner.
guewen
force-pushed
the
18.0-queue-job-channel-config
branch
from
September 12, 2026 15:22
b037b8f to
9d0b7f4
Compare
Member
Author
|
I ran 4 times this benchmark (#981) for each mode: server-side channels / per-database channels (on my laptop, performance mode) with 8 concurrent jobs on 10 workers. The throughput is close has some variance in both modes but with close results: Server-side channels:
Per-database channels (I had to exclude an outlier at 16'329 j/m I could not explain):
DB prepared with server-side runsper database channels runsBeside the throughput, the round robin seems to show good results as the queue time is almost equal for each database (of course, under the conditions of this benchmark, yet to prove in real life). |
guewen
marked this pull request as ready for review
September 12, 2026 17:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #765 (comment)
Introduction of new configuration parameters
With this in the config file, the job runner uses the static server-side configuration as before, with no hot-reload (single, shared channel manager):
With this in the config file, the job runner uses the channels configured in the database (the job runner builds one channel manager per database):
When capacity or other options of a channel (or channels are created/deleted, paused) change, the channel manager for the concerned database is hot reloaded.
Capacity can be constrained by database:
Or with patterns
When
db_max_capacityis not configured, each database gets a max capacity of the globalmax_capacity.When there is more jobs pending than
max_capacityacross the databases, a round-robin allows each database to enqueue jobs in turns.The
channelsoption for server-side configuration takes precedence over the new configuration.What does this bring?
Migration path
Since the server-side configuration (
ODOO_QUEUE_JOB_CHANNELSorchannelsin config file) is used by default, updating the addon has no effect. It allows to configure the channels capacity and other options on the UI / by script of the databases, then the server-side configuration can be replaced by the parametersmax_capacityanddb_max_capacity(or their environment variables counterparts).Caveat
When using per DB channel managers, we cannot have a shared channel for several databases since each database has its own channels (use case mentioned in #765 (comment)). It could be possible to implement it using a single channel manager that coexists with the db channel managers, yet to define the specifics, e.g. use another root (
shared:3, shared.foo:2) or a new optionshared_channels = root.sharedwhere the subchannels defined in this option are excluded from the DB channels.In the meantime, using the server-side channels instead of per-db channels is a trade-off to accept if this use case is essential for a server.