Users assigned to multiple groups experience permission errors due to cycling through service accounts on UI refresh

[perryb3693]

Pre-requisites

• I have double-checked my configuration
• I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
• I have searched existing issues and could not find a match for this bug
• I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

Description:

I'm experiencing an issue with Argo Workflows where users who are assigned to multiple groups are being cycled through the different service accounts associated with each group whenever the UI is refreshed. This is causing permission errors because specific service accounts are only able to access certain namespaces.

Here's a step-by-step reproduction of the issue:

A user is assigned to multiple groups (e.g. group1, group2, etc.)
Each group has a corresponding service account (e.g. sa-group1, sa-group2, etc.) that is used to access specific namespaces
The user logs in to the Argo Workflows UI and is initially assigned to one of the service accounts (e.g. sa-group1)
When the UI is refreshed (e.g. by clicking on a different tab or navigating back to the dashboard), the user is suddenly assigned to a different service account (e.g. sa-group2)
Because sa-group2 only has access to a different set of namespaces, the user experiences permission errors when trying to access resources that were previously accessible with sa-group1
This issue is causing significant problems for our users, as they are unable to reliably access the resources they need to do their jobs. It's also causing confusion and frustration, as the permission errors seem to be occurring randomly.

Expected behavior:

Users who are assigned to multiple groups should be able to access resources using the service account associated with the group they are currently working with, without being cycled through different service accounts on UI refresh.

Version(s)

v3.5.10

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

Steps to reproduce:

1. Assign a user to multiple groups (e.g. group1, group2, etc.)
2. Configure each group to use a different service account (e.g. sa-group1, sa-group2, etc.)
3. Log in to the Argo Workflows UI as the user and verify that they are initially assigned to one of the service accounts
4. Refresh the UI (e.g. by clicking on a different tab or navigating back to the dashboard) and verify that the user is assigned to a different service account

Logs from the workflow controller

kubectl logs -n argo argo-workflows-server
time="2025-01-27T15:23:16.818Z" level=info msg="selected SSO RBAC service account for user" email=<user> loginServiceAccount=ncti-user serviceAccount=ncti-user ssoDelegated=false ssoDelegationAllowed=false subject=22f175b3-d5e4-4165-af7c-63762a267196
time="2025-01-27T15:23:16.831Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=info.InfoService grpc.start_time="2025-01-27T15:23:16Z" grpc.time_ms=17.203 span.kind=server system=grpc
time="2025-01-27T15:23:16.832Z" level=info duration=17.937993ms method=GET path=/api/v1/userinfo size=409 status=0
time="2025-01-27T15:23:16.981Z" level=info msg="selected SSO RBAC service account for user" email=<user> loginServiceAccount=workflows-user serviceAccount=workflows-user ssoDelegated=false ssoDelegationAllowed=false subject=22f175b3-d5e4-4165-af7c-63762a267196
time="2025-01-27T15:23:16.982Z" level=info msg="selected SSO RBAC service account for user" email=<user> loginServiceAccount=darkstar-user serviceAccount=darkstar-user ssoDelegated=false ssoDelegationAllowed=false subject=22f175b3-d5e4-4165-af7c-63762a267196

This shows the workflow server assigning a new service account every time an API request is made.

Logs from in your workflow's wait container

{"code":7,"message":"workflows.argoproj.io is forbidden: User \"system:serviceaccount:argo:ncti-user\" cannot create resource \"workflows\" in API group \"argoproj.io\" in the namespace \"hf-specialized\""}: workflows.argoproj.io is forbidden: User "system:serviceaccount:argo:ncti-user" cannot create resource "workflows" in API group "argoproj.io" in the namespace "hf-specialized"

This is the error the user recieves due to being randomly assigned a different serviceaccount when attempting to create a workflow in a specific namespace.