Set custom annotations when create user workspace

[disaster37]

Describe the bug

When put this directive CHE_INFRA_KUBERNETES_NAMESPACE_ANNOTATIONS: 'che.eclipse.org/username=<username>,field.cattle.io/projectId=c-m-27mnjwcx:p-m2mjd on extraProperties, is not working.

I need to set this annotations when eclipse-che create user workspace. I know that I can pre create namespace as expected, but is not possible because of I doesn't know in advance the users wil be connect on.

Che version

7.98

Steps to reproduce

Deploy eclipse che like you want and add:

apiVersion: org.eclipse.che/v2
kind: CheCluster
spec:
  components:
    cheServer:
      extraProperties:
        CHE_INFRA_KUBERNETES_NAMESPACE_ANNOTATIONS: 'che.eclipse.org/username=<username>,field.cattle.io/projectId=c-57d7n:p-8j45h'
        CHE_INFRA_KUBERNETES_NAMESPACE_LABELS: 'app.kubernetes.io/part-of=che.eclipse.org,app.kubernetes.io/component=workspaces-namespace,field.cattle.io/projectId=p-8j45h'

Expected behavior

The namespace created must have expected annotations, aka che.eclipse.org/username=<username>,field.cattle.io/projectId=c-57d7n:p-8j45h

But, there are nothink here, not che.eclipse.org/username=<username> and not field.cattle.io/projectId=c-57d7n:p-8j45h

Runtime

other (please specify in additional context)

Screenshots

No response

Installation method

chectl/latest

Environment

Linux

Eclipse Che Logs

Additional context

No response

[ibuziuk]

@vinokurig hello, could you please take a look? Are those properties still valid?

[tolusha]

I've just checked on minikube

% oc get namespaces admin-che -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    che.eclipse.org/username: admin
    field.cattle.io/projectId: c-57d7n:p-8j45h
  creationTimestamp: "2025-02-18T08:38:30Z"
  labels:
    app.kubernetes.io/component: workspaces-namespace
    app.kubernetes.io/part-of: che.eclipse.org
    field.cattle.io/projectId: p-8j45h
    kubernetes.io/metadata.name: admin-che
  name: admin-che
  resourceVersion: "3588"
  uid: 66938dec-7742-4452-8b47-4f1d900bf154
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

[vinokurig]

Could not reproduce the issue on openshift, the labels and annotations from the extraproperties were propagated to the automatically created namespace:

kind: Namespace
apiVersion: v1
metadata:
  name: cluster-admin-che
  uid: c5bbccfa-e5c5-415e-a2b3-34d53f07aafc
  resourceVersion: '34969'
  creationTimestamp: '2025-02-18T08:42:18Z'
  labels:
    app.kubernetes.io/component: workspaces-namespace
    app.kubernetes.io/part-of: che.eclipse.org
    field.cattle.io/projectId: p-8j45h
    kubernetes.io/metadata.name: cluster-admin-che
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest
  annotations:
    che.eclipse.org/username: testuser
    field.cattle.io/projectId: 'c-57d7n:p-8j45h'
    openshift.io/description: ''
    openshift.io/display-name: ''
    openshift.io/requester: 'system:serviceaccount:eclipse-che:che'
    openshift.io/sa.scc.mcs: 's0:c29,c19'
    openshift.io/sa.scc.supplemental-groups: 1000850000/10000
    openshift.io/sa.scc.uid-range: 1000850000/10000
spec:
  finalizers:
    - kubernetes
status:
  phase: Active

[ibuziuk]

@disaster37 hello, do you have any other details? otherwise we are going to close as Can't Reproduce

[disaster37]

Hi, I use it on rancher / rke2. If I set only CHE_INFRA_KUBERNETES_NAMESPACE_ANNOTATIONS: 'che.eclipse.org/username=<username>,fu=bar it work fine. It seems there are matter with field.cattle.io/projectId= there are a way to troubleshooting what the hell on my environment?

[tolusha]

Try to set log level to DEBUG and check server logs then.

spec:
  components:
    cheServer:
      logLevel: DEBUG

All magic happens here https://github.com/eclipse-che/che-server/blob/main/infrastructures/kubernetes/src/main/java/org/eclipse/che/workspace/infrastructure/kubernetes/namespace/KubernetesNamespace.java#L218

[disaster37]

Hum, I found the following log:

2025-02-20 09:49:26,546[nio-8080-exec-1]  [WARN ] [.c.w.i.k.n.KubernetesNamespace 247]  - Can't annotate the namespace due to lack of permissions. Grant cluster-wide permissions to `get` and `update` the `namespaces` to the `che` service account (Che operator might have already prepared a cluster role called `che-namespace-editor` for this, depending on its configuration). Alternatively, consider disabling the feature by setting `che.infra.kubernetes.namepsace.annotate` to `false`.

When I look on my cluster, I found the clusterRoleBinding eclipse-che-cheworkspaces-namespaces-clusterrole that bind the service account eclipse-che/che to clusterRole eclipse-che-cheworkspaces-namespaces-clusterrole.

And the cluster role eclipse-che-cheworkspaces-namespaces-clusterrole have a permission to namespace

- apiGroups:
      - ''
    resources:
      - namespaces
    verbs:
      - get
      - create
      - update
      - list

[disaster37]

I suspect error have no direct link with RBAC. Because it work on your platform, the only way is because rancher operator add some annotations when new namespace is created. I think it break che server when it try to annotate the namespace. Maybee it first create namespace then it annotate it. But because namespace was updated between create and annotate by rancher operator, It get a forbiden because object already updated. Somethink like this...

[tolusha]

I am wondering if you are able kubectl edit namespace <..> to update it ?