bubblewrap fails to mount gitconfig if it's a symlink

[happenslol]

I'm on nixos and I'm managing my dotfiles with home-manager, which includes my ~/.gitconfig. This leads to a peculiar problem:

$ fence curl https://example.com
bwrap: Can't create file at /home/happens/.gitconfig: No such file or directory

My gitconfig is a symlink into the nix store, which is stored on a separate drive at /nix. From what I understand, bubblewrap can't bind-mount symlinks that lead to other drives, which means I can't use fence unless I allow writing to my gitignore file.

The easiest solution here would be to just exclude symlinks there, but that has security implications, of course. The cleanest solution would (imho) be to resolve symlinks to their actual path and then mount that path to the original symlink location.

[jy-tan]

Looking into this

[jy-tan]

Pushed a fix in v0.1.32. Let me know if it that works.

[happenslol]

Well, now it's onto the next one 😅

$ fence curl https://example.com
bwrap: Can't create file at /bin/chroot: Read-only file system

Haven't had the time to look at the cause for this yet, will do later

[jy-tan]

I’ve prepared a PR that canonicalizes deny mount targets during Linux mount planning (including runtime exec-deny mounts), with regression tests for both symlinked ~/.gitconfig and /bin alias cases - #60.

I don't use Nix at the moment... do you mind checking out that branch, building the binary and confirm if it works?

If it still fails, please share:

• full stderr,
• ls -l /bin/chroot,
• ls -l ~/.gitconfig,
• distro + kernel version

Thanks!

[happenslol]

I've looked into it some more, and a few more issues popped up. Nix creates a lot of sub-mounts, and bwrap's --ro-bind / / is non-recursive, so both /run (which is a tmpfs on nixos) and /nix (which is a zfs dataset for me specifically) are empty inside bwrap.

Claude came up with a function that reads /proc/self/mountinfo, gets sub-mounts of / and also ro-mounts them. That's definitely... a choice. I've also had to add /nix to the landlock allow list.

Even though I got the fence curl https://example.com to work, the tests still fail in a few places because of the way nixos sets its path up. Not sure if it's worth it to put more work into this.

If you're interested in the patches I needed I can push them, but I'm fine with just not pursuing nixos support.

[jy-tan]

Thanks for digging further. Instead doing full recursive root handling, what do you think about adding a config option like extraReadableMounts: ["/nix", "/run"] and include descent submounts only for those roots (are there a lot?) + matching Landlock read allows. Would this be helpful?

[happenslol]

That should allow me to configure it into a working state, I think. Ultimately, it might be best to not try and cover every case dynamically, and instead have some heuristics detecting things like nixos.

[jy-tan]

Opened #62, can you test against this branch and let me know if this works for you, or if I missed out any obvious guidance / features for common Nix setups. Thanks!

[happenslol]

Found 2 more issues on that branch. If I'm being honest, I'm not sure if I think this is the right fit for NixOS, or at least for me specific setup. I really haven't had the time to dig deep into it, and while I can provide you with the hacks I came up with to get the branch to work for me, I don't want to vouch for this as the correct way to solve it.

I'd rather have someone who has time to work on this look into it further than to blindly merge something that might work right now, but without any good understanding of why.

That being said, to the best of my knowledge:

• Multicall binaries such as sleep fail if we fully resolve every symlink, so if the filename changes we need to stop resolution, I think:

diff --git a/internal/sandbox/runtime_exec_deny.go b/internal/sandbox/runtime_exec_deny.go
index 9cf05e6..2b71fe7 100644
--- a/internal/sandbox/runtime_exec_deny.go
+++ b/internal/sandbox/runtime_exec_deny.go
@@ -103,6 +103,15 @@ func resolveExecutablePaths(token string) []string {
 		// Bubblewrap is strict about mounting over paths with symlink components;
 		// attempting to bind-mask /bin/foo can fail even when /usr/bin/foo exists.
 		if resolved, err := filepath.EvalSymlinks(p); err == nil && resolved != "" {
+			// If the basename changed during symlink resolution, this is a
+			// multi-call binary (e.g. chroot -> coreutils, halt -> systemctl,
+			// insmod -> kmod). Masking the multi-call target would break all
+			// other commands that share it, and bind mounts always follow
+			// symlinks so we can't selectively mask just the symlink.
+			// Skip entirely; the preflight command check still blocks these.
+			if filepath.Base(resolved) != filepath.Base(p) {
+				return
+			}
 			add(resolved)
 			return
 		}

• I get Error: failed to start command: fork/exec /run/current-system/sw/bin/sh: operation not permitted when running in a terminal, but not inside nested shells (e.g. claude code or other agents). The following fixes this, but I have no idea how reliably.

diff --git a/cmd/fence/main.go b/cmd/fence/main.go
index e896381..08ab550 100644
--- a/cmd/fence/main.go
+++ b/cmd/fence/main.go
@@ -272,7 +272,7 @@ func runCommand(cmd *cobra.Command, args []string) error {
 	//   "cannot set terminal process group: Operation not permitted"
 	//   "no job control in this shell"
 	isTTY := term.IsTerminal(int(os.Stdin.Fd()))
-	if isTTY {
+	if isTTY && !usePTY {
 		execCmd.SysProcAttr = &syscall.SysProcAttr{
 			Setpgid: true,
 			Pgid:    0, // child gets its own process group (pgid = child pid)

I don't think I'll have much more time to test things out here, sorry about raising this without having the time to really help with it.

[jy-tan]

@happenslol totally understand, thanks for investigating this far.

@podocarp has more experience with NixOS. Feel free to chime in if you have any thoughts.

[podocarp]

Wouldn't this impact more than nixos users? Maybe it's not as common as nix to have sub mounts under / for other distros but e.g. it's fairly common to mount /home and /boot separately as well, right? It's just not going to fail as hard.

[dwt]

@happenslol Do you think we could provide a nix vm test that reproduces this issue? I am thinking that it might benefit this project to have some acceptance tests like that, and it would allow us to make sure this issue does not come back.