Add emerge options to require free space/inodes in tmpdir #1353
Conversation
This adds --jobs-tmpdir-require-free-gb=GB --jobs-tmpdir-require-free-kilo-inodes=INODES as emerge emerge options. When those are used with --jobs, makes portage/emerge check that PORTAGE_TMPDIR has sufficient free resources before a new job is started. Thanks goes out to Zac Medico, as this was inspired by gentoo#1351, with the following differences: - options are absolute values, not relatives ones - defaults for both options are specified - option values are scaled, using a decaying function, considering the number or running jobs - emit a warning once a threshold is reached Note that the scaling of the resource constraints can not be perfect in the presence of concurrently running emerge jobs and without _can_add_job() being provided with the number of jobs that are potentially added. It is always possible that a emerge job has not yet used much of the filesystem when we check the remaining filesystem resources, and later on uses much more than the scaling function accounted for it. Ultimately, there is a tradeoff between portage limiting parallelism needlessly (but still being able to emerge all packages) and portage failing due to missing resources in PORTAGE_TMPDIR. The chosen defaults are rather large and most packages use much less filesystem resources then the scaling function accounts for them. Therefore, the implemented approach idea is to lean towards favoring functionality over parallelism. Bugs: https://bugs.gentoo.org/934382 Signed-off-by: Florian Schmaus <[email protected]>
| required_free_bytes = ( | ||
| self._jobs_tmpdir_require_free_gb * 1024 * 1024 * 1024 | ||
| ) | ||
| required_free_bytes = scale_to_jobs(required_free_bytes) |
There was a problem hiding this comment.
How can we be sure that this scaled required_free_bytes value stays within bounds of the file-system size?
There was a problem hiding this comment.
This is currently not ensured, although we could cap required_free_bytes and `required_free_inodes)
However, I am not sure if this is even required. Such a case is not fatal, "just" portage's emerge job parallelism is limited.
There was a problem hiding this comment.
The scaling is a significant adjustment to the user's input, so we should document it in the man page.
I suppose the scaling behavior could even be controlled by a separate option, possibly extensible with scaling plugins kind of like our metadata cache plugin framework.
Here's an idea for a --jobs-tmpdir option that can be specified multiple times in order to load multiple plugins: #1351 (comment).
| if not self._jobs_tmpdir_free_kilo_inodes: | ||
| # dev-lang/rust-1.77.1: ~ 450k inodes | ||
| # www-client/chromium-126.0.6478.57: ~1011K | ||
| self._jobs_tmpdir_free_kilo_inodes = 1100 |
There was a problem hiding this comment.
It looks like you should possibly distinguish between 0 and None so that the user can specify 0 to explicitly disable the feature.
Alternative to #1351. Seems to work as intended, but more testing is always welcomed.
This adds
--jobs-tmpdir-require-free-gb=GB
--jobs-tmpdir-require-free-kilo-inodes=INODES
as emerge emerge options.
When used with --jobs, makes portage/emerge check that PORTAGE_TMPDIR has sufficient free resources before a new job is started.
Thanks goes out to Zac Medico, as this was inspired by #1351, with the following differences:
Bugs: https://bugs.gentoo.org/934382
This seems to work as intended, but more testing is always welcomed.