ci: use fuse-overlayfs to reduce runtime storage used by builder by thunder-coding · Pull Request #28602 · termux/termux-packages

thunder-coding · 2026-02-23T14:41:02Z

For list of major changes, give a look at the first commit description of this PR, along with the titles of other commits

Will be merging this together with Python 3.13 update on ~~1st March~~ 4th March to reduce the number off docker image builds (Python 3.13 is also modifying the docker image).

thunder-coding · 2026-02-23T15:22:38Z

 else
 	REPOROOT="$(dirname $(readlink -f $0))/../"
-	SEC_OPT=" --security-opt seccomp=$REPOROOT/scripts/profile.json"
+	SEC_OPT=" --security-opt seccomp=$REPOROOT/scripts/profile.json --security-opt apparmor=_custom-termux-package-builder --cap-add CAP_SYS_ADMIN --device /dev/fuse"


@fornwall as far as I know you are the only person using Darwin for building packages locally. Does the fuse work on Darwin? If I am not mistaken, on MacOS systems, docker runs as on a lightweight VM. I'm not sure if apparmor profiles work on it. Can you confirm?

twaik · 2026-02-23T15:39:25Z

I love the idea but I have a few questions.

Will this affect regular non-docker builds on a typical home computer (random ubuntu host, after installing all required dependencies ofc) or it will break everything?
Will it break on-device building?
If the target is create standalone toolchain with minimal storage overhead we could make wrapper scripts pointing to target ndk (probably we could even cache it in ~/.termux-build instead of ~/lib) with all required flags. Also we could use lndir to create sort of copy of sysroot and replace/patch all files we need there.

TomJo2000 · 2026-02-23T16:43:55Z

Just so we're clear, this change will only require AppArmor to be installed within the container, right?
It's a fairly Debian/Ubuntu specific subsystem and is not shipped by default on most distributions not based on Debian.
So requiring AppArmor on the host would impose an undue level of complication for anyone using a distro that doesn't already make use of it.

thunder-coding · 2026-02-23T21:16:45Z

Will this affect regular non-docker builds on a typical home computer (random ubuntu host, after installing all required dependencies ofc) or it will break everything?

as long as /dev/fuse is available, it should be no issue.

Will it break on-device building?

No, NDK is only used when building outside Termux

If the target is create standalone toolchain with minimal storage overhead we could make wrapper scripts pointing to target ndk (probably we could even cache it in ~/.termux-build instead of ~/lib) with all required flags. Also we could use lndir to create sort of copy of sysroot and replace/patch all files we need there.

Could be done, I thought of that earlier before trying to think of fuse. But our buildscripts assume that we do not modify upstream NDK in ~/lib or that in $ANDROID_HOME, and thus we copy it to our own location before making our own changes to the headers and adding extra binaries which we use. We could even go as far as to ship our modified toolchain in the docker image instead of doing it on the fly for the builds, but that too would require modifying our build logic to deal with it. Additionally for some packages, we do set TERMUX_PKG_API_LEVEL, which modifies how termux_setup_toolchain_29 behaves, so I believe it is ideal to use fuse filesystem for this. This should ensure that we use minimal space for all builds.

Just so we're clear, this change will only require AppArmor to be installed within the container, right?

AppArmor works at the kernel level, it won't be able to work in monitoring the docker container if it is not running on the host.

It's a fairly Debian/Ubuntu specific subsystem and is not shipped by default on most distributions not based on Debian.

It's already packaged by Alpine, Gentoo, Guix, openSUSE, Void, etc. So I think it's fine to depend on it? Could go and add a config to not use apparmor for backward compatibility, but I believe that some of the apparmor hardening that is possible is worth it (like setuid/setgid restrictions).

So requiring AppArmor on the host would impose an undue level of complication for anyone using a distro that doesn't already make use of it.

Should be as simple as installing the apparmor package and enabling it, with their bootmanager, and doing a simple reboot. Does not need to have a distro that is making use of it, just needs to package it. Also, I don't think that disabling setuid and setgid calls is possible without apparmor just after changing the builder uid/gid. This should fully forbid sudo inside containers

Also this is the beginning of fuse stuff. If this merge doesn't break anything, I'll be further using fuse for making builds much more reproducible by mounting /data as overlayfs using fuse-overlayfs, with just the packages that are needed by the package during building. This will totally remove the need for having to deal with automagic dependencies or weird python packaging errors I've noticed that happen only when packages are built in certain order.

robertkirkman · 2026-02-23T21:22:09Z

Should be as simple as installing the apparmor package and enabling it, with their bootmanager, and doing a simple reboot. Does not need to have a distro that is making use of it, just needs to package it. Also, I don't think that disabling setuid and setgid calls is possible without apparmor just after changing the builder uid/gid. This should fully forbid sudo inside containers

Will there be an option to disable this so that sudo can be used temporarily inside the container for development and troubleshooting purpose?

I am a very heavy user of sudo inside the container. Of course, I remove instances of sudo before pushing my changes to PRs (or before merging the PR if it was a test of GitHub Actions), so they aren't seen there.

TomJo2000 · 2026-02-23T21:23:46Z

It's already packaged by Alpine, Gentoo, Guix, openSUSE, Void, etc. So I think it's fine to depend on it? Could go and add a config to not use apparmor for backward compatibility, but I believe that some of the apparmor hardening that is possible is worth it (like setuid/setgid restrictions).

I'm not saying it's not available, I'm saying asking contributors to enable a security subsystem their distro may not ship and that they may not understand well is a significant burden to contributing.
I absolutely agree that the hardening makes complete sense for our CI, but for local builds making it a requirement is a hard sell.

robertkirkman · 2026-02-23T21:25:30Z

It's already packaged by Alpine, Gentoo, Guix, openSUSE, Void, etc. So I think it's fine to depend on it? Could go and add a config to not use apparmor for backward compatibility, but I believe that some of the apparmor hardening that is possible is worth it (like setuid/setgid restrictions).

I'm not saying it's not available, I'm saying asking contributors to enable a security subsystem their distro may not ship and that they may not understand well is a significant burden to contributing. I absolutely agree that the hardening makes complete sense for our CI, but for local builds making it a requirement is a hard sell.

I use Gentoo, and Gentoo does have an apparmor package, but if possible I would prefer not to have to install it or have to use a VM with it installed, to continue developing Termux packages.

thunder-coding · 2026-02-23T21:26:33Z

Will there be an option to disable this so that sudo can be used temporarily inside the container for development and troubleshooting purpose?

You can go and load the relaxed profile using cat ./scripts/profile-relaxed.apparmor | sed -e "s/{{CONTAINER_NAME}}/$CONTAINER_NAME/g" | $SUDO apparmor_parser -rK. I'll add them as utility scripts in ./scripts/bin/ for you (and me). I too modify the builder environment as well to make things easier for patching on the live builds at times :P

I absolutely agree that the hardening makes complete sense for our CI, but for local builds making it a requirement is a hard sell.

Thanks, makes sense. Will be making apparmor optional then!

TomJo2000 · 2026-02-23T21:27:18Z

I am a very heavy user of sudo inside the container. Of course, I remove instances of sudo before pushing my changes to PRs (or before merging the PR if it was a test of GitHub Actions), so they aren't seen there.

Being open about our motivations is something that can definitely help avoid misunderstandings.
I don't use AppArmor on my system, and I have no desire to start using AppArmor on my system, I understand the benefits it provides for hardening a production environment but I do not see a reason to use it on my personal machine.

thunder-coding · 2026-02-23T21:35:03Z

I use Gentoo, and Gentoo does have an apparmor package, but if possible I would prefer not to have to install it or have to use a VM with it installed, to continue developing Termux packages.

apparmor wasn't needed earlier, but will be required (or let's say highly recommended) if we continue to go with fuse-based approach for Android NDK and my future plans for making builds more reproducible. mount syscalls on Linux do require CAP_SYS_ADMIN capability which is a dangerous capability. The kernel docs do state that passing in mount permissions and /dev/fuse to containers should be fine mostly as long as things are configured correctly: https://docs.kernel.org/next/filesystems/fuse-passthrough.html. So likely this requirement too might be relaxed with newer kernel releases. (See torvalds/linux@18ee43c, it's just 6 months ago as of writing this comment so might take a while to actually happen)

And the setuid, setgid changes are being made just because let's just do this while we are at it instead of waiting for another occassion

robertkirkman · 2026-02-23T21:42:26Z

I'll be further using fuse for making builds much more reproducible by mounting /data as overlayfs using fuse-overlayfs, with just the packages that are needed by the package during building.

In case you wonder, both MrAdityaAlok (in an earlier version of buildorder.py changes PR) and fornwall (in Google Play Termux, visible currently) have implemented their own versions of that, but both of them unfortunately have a somewhat inconvenient performance overhead.

Yours is the first version of the idea I have heard that sounds to me like it might solve the performance overhead, since it does sound to me like nested fuse-overlayfs is probably a lot faster than actually copying files around, but if you wonder how yours compares to previous attempts, you can look at Google Play Termux termux-packages and consider benchmarking in comparison to it.

If your method ends up having extremely low performance overhead to the point that it's clearly about the same speed as the current performance, then benchmarking likely isn't necessary.

robertkirkman · 2026-02-23T21:57:09Z

Will it break on-device building?

In the long term, if the follow-up change to /data folder is implemented and kept, it will slowly degrade the compatibility of packages with on-device building by diverging the behavior of cross-compilations further from on-device builds more than they currently are, allowing more packages to pass CI without errors that persist in the on-device build than currently do.

however, that problem can be resolved (and improved to catch more errors that are currently not caught) if this PR is approved, because then a workflow that tests on-device builds could eventually be added to the main repository.

deblobbify, replacing the system folder with several other components termux-docker#72

robertkirkman · 2026-02-24T05:34:47Z

When MrAdityaAlok made their space optimization for the CI, they posted a detailed result of how much space would be saved, here is what they showed,

ci(packages): Move space cleaning logic to separate script #23875 (comment)

I found that very helpful, especially for thinking about how much additional space I could safely use during build while designing contributions,

if it's not too much trouble, could you run a similar comparison in GitHub Actions for this space optimization?

thunder-coding · 2026-02-27T12:32:46Z

if it's not too much trouble, could you run a similar comparison in GitHub Actions for this space optimization?

Didn't do comparision for this PR, but I have seen locally a decrease of ~2.4G in container sizes. So combined with #28627, we should be seeing 36G+2.4G (for the free-space.sh core part + 2.4G is for removal of compressed docker image) + additional 2.4G for the uncompressed copy of the entire NDK. For building termux-exec, there will be an additional 2.4G decrease in size due to it setting up another toolchain for building of some test executables.

Would appreciate some reviews on this as I am planning to merge this PR #28627 as well as #27739 together

Earlier 54G -> 25G Now 53G -> 16G (10G excluded for docker image)

Does this work? Yes Is this cursed? Not as cursed as using bash for build system

This should allow for more flexible configuration of docker containers by allowing devs to pass on their own flags to docker Also ~/clean.sh will now not remove /home/builder/.termux-build to account for cases where /home/builder/.termux-build is a volume mounted to the docker image, where it is not possible to remove the directory

We now are always using docker for builds, so can't do this

Do not clutter what we do in "Gather build summary" step

Also make sure we use this optimization in package_updates.yml

~/.termux-build on host Should be helpful for local builds for using IDEs and host tools for development

In commits, %ci:free-space will force freeing space in commits In workflow dispatch, a new checkbox should be available

Instead of only supporting one of the flags, we now support passing multiple flags at the same time for more convenience. The command line argument parser will exit as soon as it detects an argument/flag it doesn't handle to preserve maximum compatibility with existing commands

rid of long length variable name

Logic error found by GitHub Copilot

**BREAKING CHANGES** This now requires AppArmor to be installed and running with docker for limiting the capabilities that `CAP_SYS_ADMIN` provides to containers. Host kernel must support fuse. The host /dev/fuse device is passed onto the container. **DETAILED DESCRIPTION** ./scripts/run-docker.sh first starts with relaxed profile, and then after changing the uid and gid of the builder user and group, drops to restricted profile. Each container get's it's own profile so that if ./scripts/run-docker.sh is run parallel with multiple containers, there is no race condition for the when we are changing the builder uid/gid, where the other container will run with higher privileges than needed. For ensuring least privileges, only mount and umount2 syscalls have been permitted in seccomp profile. Additionally rules for allowing clone, clone3 and similar syscalls when certain contain conditions are met and only when CAP_SYS_ADMIN is not set have been removed as we aren't allowing these syscalls when CAP_SYS_ADMIN is set. The AppArmor profile is based on Docker's default AppArmor profile. The profile was extracted using nerdctl (which is an alternate CLI interface to Docker CLI). The profile can be extracted using `nerdctl apparmor inspect`. There are two AppArmor profiles we have setup, one restricted and relaxed. Currently there is little difference between relaxed and restricted profile. The only difference is that relaxed profile allows any kind of mount syscall, while the restricted profile only allows mount syscalls only for fuse.fuse-overlayfs Regarding security of passing /dev/fuse to containers, the Linux kernel documentation specifies that it should be fine to pass this to namespaces. The CAP_SYSTEM_ADMIN is needed only for the mount syscall to work. Due to some historic reasons this needs this dangerous capability. Although the syscall is needed we are only allowing mount and umount syscalls to happen inside the container, so seccomp profile and apparmor profile should be doing the damage control. Linux kernel documentation for fuse-passthrough: https://docs.kernel.org/6.16/filesystems/fuse-passthrough.html Upstream Linux kernel commit for fuse documentation: torvalds/linux@18ee43c Even without apparmor, things should be fine as we aren't fiddling around much with apparmor for security reasons, just currently limiting where the fuse filesystem can be used. In future, AppArmor profiles can also be used for further hardening of docker image

@licy183

@licy183 and me for Seccomp profile Me for apparmor profile Feel free to add yourself if you believe that you can deal with this nicely. Mostly for others looking for maintenance of the apparmor profile. Just grab the docker's default config using nerdctl apparmor inspect, diff with the current config and figure it out For the seccomp profile, just diff with the exact commit of moby/moby's seccomp profile and store the updated JSON

AppArmor isn't configured by default on distributions other than Ubuntu, so don't mandate it. AppArmor proper configuration and setup is a huge pain especially if you aren't familiar with it and containers in general. Even a lot of the maintainers aren't already familiar with it and using it already so let's just keep it optional and do not use it if not detected on host.

only respect TERMUX_DOCKER_USE_SUDO for running docker commands Even if docker is setup in rootless mode, apparmor needs to be run with sudo. This was the cause of CI failure and me trying to bang my head on why things weren't working on CI. Thanks, a "sudo" is all it takes. I wish it worked in real life as well on people

We assume in a lot of places that all files in termux-packages are trusted. So make sure that the container isn't able to modify anything in this environment. Also do not allow any changes to ~/lib/ where we are storing Android NDK and SDK.

not installeed

thunder-coding · 2026-03-03T14:54:58Z

Manually merged directly in master. I don't know why it didn't got detected by GitHub, probably I messed up somewhere

thunder-coding requested a review from Grimler91 as a code owner February 23, 2026 14:41

thunder-coding requested review from TomJo2000, fornwall, licy183 and robertkirkman February 23, 2026 14:41

thunder-coding force-pushed the android-ndk-overlayfs branch from a7ddb84 to 4235b7c Compare February 23, 2026 14:46

thunder-coding self-assigned this Feb 23, 2026

thunder-coding added this to @thunder-coding's public tracker Feb 23, 2026

thunder-coding commented Feb 23, 2026

View reviewed changes

thunder-coding marked this pull request as draft February 23, 2026 21:26

robertkirkman reviewed Feb 24, 2026

View reviewed changes

Comment thread clean.sh Outdated

thunder-coding force-pushed the android-ndk-overlayfs branch 4 times, most recently from 029595e to 890f91a Compare February 25, 2026 19:10

thunder-coding marked this pull request as ready for review February 25, 2026 19:11

thunder-coding mentioned this pull request Feb 25, 2026

ci: free more disk space #28627

Merged

thunder-coding force-pushed the android-ndk-overlayfs branch 2 times, most recently from e42bf19 to 114cd78 Compare March 3, 2026 14:01

thunder-coding added 19 commits March 3, 2026 20:15

scripts(free-space.sh): remove more stuff from runner

e4e0042

Earlier 54G -> 25G Now 53G -> 16G (10G excluded for docker image)

ci: also clean docker compressed archives

a7cbd77

Does this work? Yes Is this cursed? Not as cursed as using bash for build system

ci(packages): do all builds in docker

002e5a3

ci(packages): do not prune docker

ee6b87c

We now are always using docker for builds, so can't do this

ci(packages): split docker image building to separate step

eef0cc5

Do not clutter what we do in "Gather build summary" step

ci: move clean compressed docker images to free-space.sh

524c773

Also make sure we use this optimization in package_updates.yml

scripts(run-docker.sh): add short option to mount /data and

a4da392

~/.termux-build on host Should be helpful for local builds for using IDEs and host tools for development

chore,ci(packages): Allow forcing free space

f133bcd

In commits, %ci:free-space will force freeing space in commits In workflow dispatch, a new checkbox should be available

chore,scripts(run-docker.sh): limit line width to 80 for usage and get

91b08c7

rid of long length variable name

fix,scripts(run-docker.sh): do not bail out on first flag

5903ad4

Logic error found by GitHub Copilot

scripts: more apparmor hardening

49eacaf

We assume in a lot of places that all files in termux-packages are trusted. So make sure that the container isn't able to modify anything in this environment. Also do not allow any changes to ~/lib/ where we are storing Android NDK and SDK.

fix,scripts(run-docker.sh): Properly handle the case when apparmor is

59ae7be

not installeed

fix,scripts(run-docker.sh): only show AppArmor log when using apparmor

758b230

thunder-coding force-pushed the android-ndk-overlayfs branch from 114cd78 to 758b230 Compare March 3, 2026 14:45

thunder-coding closed this Mar 3, 2026

github-project-automation bot moved this to Done in @thunder-coding's public tracker Mar 3, 2026

thunder-coding mentioned this pull request Mar 4, 2026

addpkg(main/turborepo): 2.8.13 #28682

Closed

robertkirkman mentioned this pull request Mar 5, 2026

chore(common-files/building-system-patches): remove do-not-use-python-from-usr-bin.patch termux-user-repository/tur#2296

Merged

thunder-coding mentioned this pull request Mar 5, 2026

fix(scripts/run-docker): remove duplicated TERMUX_BUILDER_IMAGE_NAME and CONTAINER_NAME #28778

Merged

This was referenced Mar 27, 2026

[Bug]: termux-am fails to build after the enablement of AppArmor #29118

Open

[Bug]: frustrating AppArmor-related error user builder is currently used by process 1 termux-user-repository/tur#2394

Closed

Uh oh!

Conversation

thunder-coding commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thunder-coding Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

twaik commented Feb 23, 2026

Uh oh!

TomJo2000 commented Feb 23, 2026

Uh oh!

thunder-coding commented Feb 23, 2026

Uh oh!

robertkirkman commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomJo2000 commented Feb 23, 2026

Uh oh!

robertkirkman commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thunder-coding commented Feb 23, 2026

Uh oh!

TomJo2000 commented Feb 23, 2026

Uh oh!

thunder-coding commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertkirkman commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertkirkman commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

robertkirkman commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thunder-coding commented Feb 27, 2026

Uh oh!

thunder-coding commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

thunder-coding commented Feb 23, 2026 •

edited

Loading

robertkirkman commented Feb 23, 2026 •

edited

Loading

robertkirkman commented Feb 23, 2026 •

edited

Loading

thunder-coding commented Feb 23, 2026 •

edited

Loading

robertkirkman commented Feb 23, 2026 •

edited

Loading

robertkirkman commented Feb 23, 2026 •

edited

Loading

robertkirkman commented Feb 24, 2026 •

edited

Loading