Support for SELinux

[SteVwonder]

Based on our internal conversation today, it appears that SELinux support is basically mandatory for DOD at this point, and the DOE may soon follow. It is unclear exactly what the entire scope of "full support" is.

A good starting point would be for Flux to launch batch jobs confined to a specific SELinux context. Initially, it might be easiest to have the target SELinux context provided by MUNGE (i.e., the user already has to be within the target context when they submit the job), but I can imagine a scenario where the user wants to be in context A and submit a job that will run in context B (assuming the user has the proper privileges to run in context B).

One potential "gotcha" that @grondo mentioned: if you aren't careful with permissions, then it is possible for the RM to create a file descriptor for a job that (once running under the target context) the job is unable to access.

[grondo]

This use case reminded me that the IMP should probably support calling out to a session PAM stack. There is already a pam_selinux.so module which supports setting a correct user context on login, so presumably we could just add this to the IMP's PAM stack and get at least a default per-user context established for jobs (at least those transitioning UIDs).

Long term, storing any current security context in a MUNGE credential is the preferred solution. To support a user submitting a job to run in a different context than they are currently in would mean we'd have to duplicate pam_selinux and parse a config file to determine the contexts to which a user has access. It would be preferable not to have to do that, as it seems perilous. Perhaps it isn't too onerous to have users switch to the security context in which they would like jobs to run before submission?

[dongahn]

Perhaps it isn't too onerous to have users switch to the security context in which they would like jobs to run before submission?

Yeah I believe this is the main use case GS folks said we need to support.

[SteVwonder]

Tagging @cg9468

[cg9468]

Users will need the ability to run jobs as their normal SELinux user context as well as when they switch to a project context. Role Base Access Control is used to assign a user to multiple projects. When a user wants to work in a project context they run a script that runs the 'newrole' command in the background which asks for their password. Once authenticated, a new xterm window opens in the users GNOME session in the proper project context so the users can perform work and submit jobs. The user can switch to multiple projects at a time if they choose. They'll have an xterm window per project, which will have the proper context assigned. The job scheduler should be able to pickup what SELinux context the job is being submitted from and run the job as that context which should persist until the job is complete. Each project context is siloed to designated areas on the filesystem, which means the output of a job has to be written into the siloed areas with the matching SELinux context only.

[garlick]

Not knowing much about SELinux, this may be a dumb question: if I enter an SELinux role by typing a password, and then submit a job, and my role is recorded in a MUNGE cred at job submit time which allows Flux to launch a job in the future in that role, is there a way for Flux to know if my access to the role has been revoked between submission and launch?

(Note that a job can submit another job with a new MUNGE cred, so potentially a user could chain jobs to keep hold of a revoked role for longer than a single job TTL, if there was no way to re-check role "membership")

[SteVwonder]

Tagging @stevedlawrence too.

[stevedlawrence]

@garlick The way this use case works, revoking a users access to a role requires an update to the SELinux policy, wich would make the context no longer valid. When Flux attempts to launch a job in that now invalid context (via setexeccon(3)), it will fail with an error code. Flux would be expected to check this error and cancel the job.

[garlick]

Thanks! That sounds like a non-problem then.

[grondo]

Tagging @dun as author of MUNGE

[garlick]

@stevedlawrence - does the context need to be validated at job submission time, e.g. by getpeercon() in a trusted part of the code, or is it safe to let the submission program running as the user provide it, and then expect that a combination of setuid() and setexeccon() at exec time will fail if the users don't match, such as if I obtained someone else's context string and provided it as my own?

[stevedlawrence]

Validation of the context return from getprevcon()/getpidcon() shouldn't be necessary. If those return without an error then it is a valid context. The core requirement is that whatever calls either of those functions needs to be trusted to securely get that information and transmit it to slurm. For example, we cannot trus the user (or the user mimicking slurm) to provide that information.

My understanding is that Munge has some capabilities to perform these kinds of actions in a secure manner, though the specifics of how I am not too familiar with.

[garlick]

MUNGE doesn't currently have those capabilities. See dun/munge#70.

I think you answered my question - I wasn't sure why we needed trusted code to collect the context if the context could only be used by the user it was issued to at exec time.