Allow to manage revision of foreign packages

[noctux]

Thank you for the nice project. It has definitively helped to keep my Arch system clean as far as possible.

Recently I've started to use aconfmgr on two systems, and sure enough it will install all my configured packages on both hosts. However, for the Foreign packages, this leaves a bit of an uneasy feeling: They do not receive the same level of vetting as a package of an TU in the official repositories recieves. And even if I trust a specific author, I will not get notified when the package maintainer on the AUR changes, etc.pp. creating a trust/security issue when aconfmgr automatically installs the latest version. So it would be nice to be able to pin foreign packages to a version that I have manually reviewed on any of my systems.

As the AUR uses git nowadays, it might be as easy as including the respective git revision alongside the package within the installation line.

Caveat:
Looking at the code, though, while this should be easily doable with the default, manual makepkg-based installation routine, it will not work for the third party-managers that easily.

I know that it's more in the projects spirit to move such enhancements to custom helpers, even though I personally might file it more as a 'security enhancement', which might be of interest. But maybe, even if this is not integrated into the main helpers, maybe there will be some interesting insights into an implementation. A potential implementation idea I have revolves around tracking the pkgbuild-repos as submodules and building from there. That way, they could be updated with a recursive git-pull, reviewed with a diff, and meanwhile we would guarantee a consistent system state with regards to AUR-package versions.

Thank you for your consideration.

[CyberShadow]

Thank you for filing this. The concern is valid and your analysis is correct.

The topic of recording or pinning package versions has come up before, with a similar complication - pacman (unlike some other distributions' package managers) does not provide a way to specify a package version to install, and will always install the latest version in its cached copy of sync repositories. In case of AUR packages, there is an additional complication: there is no way to know which revision of a package recipe (PKGBUILD and accompanying files) was used to build the package file (.pkg.tar.xz). As a result, aconfmgr save doesn't have a way to record the revision of AUR packages.

This is not insurmountable: conceivably, aconfmgr could track which revision results in which .pkg.tar.xz as it's installing and building packages. This could even be done when using AUR helpers, by peeking into their cache directories and looking at the revision of the cloned AUR package. Some AUR helpers delete the package file after installing it, so we can use a combination of pkgver and package build date to identify packages. So, this could be done by having aconfmgr apply record these associations when installing packages, then suggest running aconfmgr save immediately afterwards to record the package versions back to the configuration.

Which would cover saving. As for applying - from a quick look, indeed none of the AUR helpers currently supported by aconfmgr support specifying an AUR revision to install. Though pinning could be implemented in the makepkg AUR handler as you've pointed out, I'm not sure this would be a notable accomplishment, considering that it is really mainly meant to be a fallback mechanism; most users will likely install and prefer using an AUR helper of their choice.

Instead, perhaps we could take an alternative approach to solve the same underlying issue. I'm not sure if it's actually more useful to prefer installing an older version of a package: this is certainly useful from the point of view of reproducibility (i.e. have the same configuration always generate exactly the same system); however, from a security perspective, this might actually be harmful due to older software having vulnerabilities. In any case, the reproducibility goal is already blocked by pacman's inability to install specific package versions; and, if there is an ongoing problem with an AUR package which warrants pinning it to a specific version, probably forking it is the more correct solution.

Possibly a more useful approach is simply to make users aware that the package version they're about to install is not the version they've last reviewed. Tracking which version that is could be done within or outside the configuration. AUR helpers generally already do this, so, if the versions are to be tracked within the configuration, that would require some way for aconfmgr to sync the last version seen by the user with the AUR helper (which would probably involve poking inside their private directories).

Finally, a much simpler and pragmatic approach would be to let AUR helpers track user-reviewed versions in their own way, and move the onus on the user if they want to somehow additionally synchronize that to avoid reviewing the same changes more than once. The built-in makepkg handler can be improved to have some simple way of reviewing changes as well. Also, there is an argument to be made regarding the overhead of reviewing the same AUR package changes when managing more than once Arch system: although synchronizing the last reviewed packages would save the user from that work, it doesn't save the cost of actually building these packages, so, users who manage more than one Arch system and use many AUR packages may want to switch to a pacman repository based management of AUR packages (regardless whether they use aconfmgr or not), such as using aurutils.

What do you think?

[noctux]

Thank you for your fast and thoughtful response.

On Tue, Apr 23, 2019 at 12:47:40PM -0700, Vladimir Panteleev wrote: The topic of recording or pinning package versions has come up before, with a similar complication - pacman (unlike some other distributions' package managers) does not provide a way to specify a package version to install, and will always install the latest version in its cached copy of sync repositories. In case of AUR packages, there is an additional complication: there is no way to know which revision of a package recipe (PKGBUILD and accompanying files) was used to build the package file (.pkg.tar.xz). As a result, `aconfmgr save` doesn't have a way to record the revision of AUR packages.

Well, afaik, this is solved for packages build with newer makepkg/pacman releases (the wiki [0] names pacman 5.1 or newer). These versions feature a `.BUILDINFO` file, which also stores a sha256sum of the PKGBUILD. As the PKGBUILD in return stores hashes of the dependencies, this might be okay (there could always be a `SKIP` directive, but one has to start somewhere...). For instance: ```bash $ acat /var/cache/pacman/pkg/ansible-2.4.2.0-1-any.pkg.tar.xz .BUILDINFO | grep pkgbuild pkgbuild_sha256sum = 7a076ee4d785e095753bc44d2a14b344e955fad975cc4b527b71124bb204b5d9 ``` However, I don't exactly know whether and where this file ends up during installation. But we could always extract it from the package cache where possible (and maybe ask nicely on the bugtracker whether the file could be copied during extraction, e.g. to `/var/lib/pacman/local/` just like pkgdesc and mtree are as well).

This is not insurmountable: conceivably, aconfmgr could track which revision results in which `.pkg.tar.xz` as it's installing and building packages. This could even be done when using AUR helpers, by peeking into their cache directories and looking at the revision of the cloned AUR package. Some AUR helpers delete the package file after installing it, so we can use a combination of pkgver and package build date to identify packages. So, this could be done by having `aconfmgr apply` record these associations when installing packages, then suggest running `aconfmgr save` immediately afterwards to record the package versions back to the configuration.

Build date might again be problematic for locally build packages. But I get your point :) In the general, benevolent case, we should actually be fine with the pkgver+pkgrel, most maintainers increment them when updating. However, this will not really help against an malevolent actor trying to distribute malware, etc. as outlined in my initial statement.

I'm not sure if it's actually more useful to prefer installing an older version of a package: [...] from a security perspective, this might actually be harmful due to older software having vulnerabilities. [...] Possibly a more useful approach is simply to make users aware that the package version they're about to install is not the version they've last reviewed.

You are definitely right, warning the user is all it should take. Especially as everyone has her or his favourite package-manager with diffing utility, etc.pp.. Yet, having an automatic way to import this review back into my config with `aconfmgr save` would make this all a whole lot more pleasing. Something like the sha256sum of the PKGBUILD mentioned above should suffice there.

Also, there is an argument to be made regarding the overhead of reviewing the same AUR package changes when managing more than once Arch system: although synchronizing the last reviewed packages would save the user from that work, it doesn't save the cost of actually building these packages.

Ideally, we could get rid of the rebuilding, when we can get a checksum that be can validated without building, that is by computation on the build-inputs. However, even if this is not possible: I don't really mind starting aconfmgr and taking a break to leave my box recompile the required packages, as long as I can do so with a peace of mind. But this might just be me, with my two boxes and a couple of ForeignPackages, speaking :)

users who manage more than one Arch system and use many AUR packages may want to switch to a pacman repository based management of AUR packages (regardless whether they use aconfmgr or not), such as using aurutils.

My gut feeling would be that people running that many boxes that maintaining their own repo is worth the effort are not the primary audience for aconfmgr (whose primary selling point of me is the ease of use in terms of reimporting changes in the configuration) as compared to more fully fledged solutions such as puppet, but I might be wrong there. Regards, Simon 0: https://wiki.archlinux.org/index.php/creating_packages

…

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #42 (comment)

[CyberShadow]

Well, afaik, this is solved for packages build with newer makepkg/pacman
releases (the wiki [0] names pacman 5.1 or newer). These versions
feature a .BUILDINFO file, which also stores a sha256sum of the
PKGBUILD. As the PKGBUILD in return stores hashes of the dependencies,
this might be okay (there could always be a SKIP directive, but one
has to start somewhere...).

For instance:

$ acat /var/cache/pacman/pkg/ansible-2.4.2.0-1-any.pkg.tar.xz .BUILDINFO | grep pkgbuild
pkgbuild_sha256sum = 7a076ee4d785e095753bc44d2a14b344e955fad975cc4b527b71124bb204b5d9

However, I don't exactly know whether and where this file ends up during
installation. But we could always extract it from the package cache
where possible (and maybe ask nicely on the bugtracker whether the file
could be copied during extraction, e.g. to /var/lib/pacman/local/ just
like pkgdesc and mtree are as well).

The problem is that this file is in the .pkg.tar.xz, which is not always available. pacman does not keep a copy of the package file, and some AUR helpers will delete it after installing it. /var/cache/pacman/pkg only contains package files from sync repos - AUR packages will not be there.

However, I checked and it looks like .BUILDINFO is included in the mtree file, which is kept by pacman. Its hash there could be used for identification.

You are definitely right, warning the user is all it should take.
Especially as everyone has her or his favourite package-manager with
diffing utility, etc.pp.. Yet, having an automatic way to import this
review back into my config with aconfmgr save would make this all a
whole lot more pleasing. Something like the sha256sum of the PKGBUILD
mentioned above should suffice there.

Well, importing the review is one half of the problem; the other is using it.

Making aconfmgr compare the latest version and then running the AUR helpers to install the latest version is not really a solution, as it creates a TOCTOU window during which a package may be updated.

Ideally, we could get rid of the rebuilding, when we can get a checksum
that be can validated without building, that is by computation on the
build-inputs. However, even if this is not possible: I don't really mind
starting aconfmgr and taking a break to leave my box recompile the
required packages, as long as I can do so with a peace of mind.

This makes sense; but, so far, I see an obvious way to implement this only for the makepkg AUR handler.

My gut feeling would be that people running that many boxes that
maintaining their own repo is worth the effort are not the primary
audience for aconfmgr (whose primary selling point of me is the ease of
use in terms of reimporting changes in the configuration) as compared to
more fully fledged solutions such as puppet, but I might be wrong there.

Of course, to each their own; I switched to aurutils after setting up the third Arch system (PC in another country).

Also, inspecting and configuring remote hosts is a feature I'd like to add to aconfmgr in the future.

---

As far as this topic is concerned, I think incremental improvement would be the way to go here. I see a few steps:

1. Make aconfmgr communicate clearly what it's doing with regards to AUR packages, and which actions are "safe" or not.

2. Make aconfmgr track changes to AUR packages it's building using makepkg. The location used to save reviewed commits could then be configured or symlinked to be under the user's configuration, thus making it shared.

3. Maybe, extend the reviewed commit tracking to support integration with AUR helpers, so users of AUR helpers don't need to review changes more than once as well. This could be done by messing with their internal directories, or perhaps we can ask the helpers' maintainers to add an official supported mechanism that aconfmgr could use.

Sounds good?

[noctux]

On Tue, Apr 23, 2019 at 03:41:33PM -0700, Vladimir Panteleev wrote: However, I checked and it looks like `.BUILDINFO` is included in the mtree file, which is kept by pacman. Its hash there could be used for identification.

I fear it is not that easy. As far as I can tell, `.BUILDINFO` also contains information about any packages (and their versions) that were installed while building the package, so the hash of `.BUILDINFO` stored in the `mtree` should be quite useless to us.

Making aconfmgr compare the latest version and then running the AUR helpers to install the latest version is not really a solution, as it creates a TOCTOU window during which a package may be updated.

IIRC aur4 rejects force pushs, so we could improve on the situation a bit if we could convince the managers to just create the package without installing, by checking the revision before *and* after the build. However, even then: if the PKGBUILD is modified in between, I have still executed some random bash script beforehand :/

Also, inspecting and configuring remote hosts is a feature I'd like to add to aconfmgr in the future.

Nice, looking forward to it :)

--- As far as this topic is concerned, I think incremental improvement would be the way to go here. I see a few steps: 1. Make aconfmgr communicate clearly what it's doing with regards to AUR packages, and which actions are "safe" or not. 2. Make aconfmgr track changes to AUR packages it's building using makepkg. The location used to save reviewed commits could then be configured or symlinked to be under the user's configuration, thus making it shared. 3. Maybe, extend the reviewed commit tracking to support integration with AUR helpers, so users of AUR helpers don't need to review changes more than once as well. This could be done by messing with their internal directories, or perhaps we can ask the helpers' maintainers to add an official supported mechanism that aconfmgr could use. Sounds good?

The plan sounds quite reasonable, so lets see how we can get it done! So step 1.1 would be to find a good classifier for what is "safe"... However, I'm struggling with the clear separation to 2., as this classifier would probably entail finding a checksumming mechanism to detect known packages? Otherwise I'd assume that every invoke to makepkg is "unsafe". ad 3.: Maybe initially asking the pacman-dev mailinglist might be a better option, as a good mechanism within pacman/makepkg might ease/eliminate a lot of redundancy with the packagemanagers (e.g. an installed .BUILDINFO file).

…

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #42 (comment)

[noctux]

ad 3.: Maybe initially asking the pacman-dev mailinglist might be a better option, as a good mechanism within pacman/makepkg might ease/eliminate a lot of redundancy with the packagemanagers (e.g. an installed .BUILDINFO file).

Oh, but if we do this, we should probably first think about what we really really want... From the top of my head: .BUILDINFO is although about reproduceability, so providing a hash of any listed sourcefile (even those marked as "SKIP") should increase both the reproduceability of builds as well as the "security" of what we want to do.