Patch to support auditd loginuid immutability #478
Conversation
1) enable quiet login by removing /etc/update-motd.d/10-uname and
truncating /etc/motd
2) modify PAM sshd session setting making pam_loginuid module optional
|
This is what the symptom looks like from Jenkins adding an SSH agent:
|
|
After successfully applying patch and re-launching docker-ssh-agent, our Jenkins SSH Agent is online: |
|
Tested on current weekly: |
|
I wasn't able to duplicate the issue from a Debian 12 computer. I don't object to the change, but would like to confirm that it is an issue that I can duplicate before the change is merged. Any suggestions of the mistake I might be making? Steps that I took while trying to duplicate the issue:
The ssh connection worked and the agent connected to Jenkins as expected. |
|
Interesting... I wonder if the PAM issue is specific to the ppc64le debian-slim docker image?
|
|
This build test was on Ubuntu 24.04.1 LTS for ppc64le using docker-ce v27.4.1 and github.com/docker/buildx v0.19.3 48d6a39 |
There was a problem hiding this comment.
I've seen a frequent recommendation to reduce the number of layers in the container image. Since those two RUN commands are part of the SSH configuration of the container image, could you attach them to the preceding command that performs SSH configuration? That way we don't add two very small layers to the image.
|
Done!
|
|
Mark, I narrowed down the issue to be related to if your Docker platform has CAP_AUDIT_WRITE capability or not. If Docker removes the audit capability of the container, then the PAM module "pam_loginuid" breaks login because its purpose is to assign the audit attribute on the process.
|
Interesting, good catch! For the sake of the general security, I believe we should set the test harness to test with this capability dropped, and then fix the PAM stuff like you did. Medium term, we should test that the image can run with both |
|
Damien, I got it to work with When I tried with @MarkEWaite, what version of docker-ce are you running? Reference: Ken
|
Docker version 27.4.1, build b9d17ea |
|
Damien, thank you for the rsyncd reference Dockerfile. I like the way you handled the motd issue better there. I am going to update this pull request with that method and also test running with all capabilities removed.
|
more elegant way the following: 1) Quiet login: disabling motd in PAM 2) enabling dropping of privileges in container: remove the requirement for CAP_AUDIT_CONTROL in PAM for SSH login
|
Report for running the following Jenkins containers with docker-inbound-agent: docker-ssh-agent: (For reference, the default capabilities that remain disabled are docker (controller): |
|
I'm still unable to determine what's really different between Docker running on ppc64le vs amd64. Note that the issue only happens on real ppc64le hardware: I'm testing on a RHEL9 POWER10 host, same exact build procedure running docker inside of a Podman Ubuntu container works without the patch for ppc64le on top of QEMU user-static on my WSL2 RHEL9 x86_64 host. I also sanity tested amd64 on x86_64 to rule-out QEMU. The only difference I came up with is the Docker control group namespace is set to The actual Linux kernel capabilities enabled themselves are identical on both the POWER10 host and x86_64 host: |
|
My question is: why doesn't the x86_64 implementation execute these pluggable authentication modules? Maybe the power implementation is correctly running these as they should. It's interesting the rsyncd container also needed to suppress motd in pam.
|
|
I have confirmed the same issue happens on x86_64 Fedora running Docker inside Podman. This means the condition of an immutable loginuid is caused by either Podman itself or a Red Hat setting. Will need to investigate further where exactly but I am concluding it is worth supporting systems that have immutable loginuid for security reasons with this patch. |
|
Here is some documentation on making loginuid immutable for security best-practices sake: |
|
Also worth noting that until recently Docker used to document that setting pam loginuid as optional was required to get containerized sshd to work. Not sure why they removed that doc article when it is still the case for those who make loginuid immutable, and the only other way would be to grant CAP_AUTH_CONTROL which is a security vulnerability: |
ksalerno99
changed the title
|
For the purposes of completing the documentation here, this is the error message one sees in docker logs that indicate your security settings make loginuid immutable: Inside the docker container docker-ssh-agent, we see that loginuid is immutable: Here is the expected behavior when we have CAP_AUDIT_CONTROL added to the container to give it elevated privileges: So we see that by adding CAP_AUDIT_CONTROL, which is a security vulnerability, we can now edit /proc/self/loginuid, and the agent deploys as expected in Jenkins CI: |
Testing done
Before making change, PAM would immediately terminate an SSH attempt to the container (RC=254):
The second issue was Jenkins authentication via SSH expects a quiet login, and even though we set "PrintMotd no" in /etc/ssh/sshd_config, /etc/pam.d/sshd is overriding this with "session optional pam_motd.so motd=/run/motd.dynamic"
Submitter checklist