Cannot set blockOwnerDeletion

[cyberox]

Describe the bug
I would like to run moco on our openshift cluster. The operator is installed correctly.
But during the creation of a MySQLCluster resource, I receive the following error:

LAST SEEN   TYPE      REASON         OBJECT                         MESSAGE
4s          Normal    NoPods         poddisruptionbudget/moco-ngw   No matching pods found
2s          Warning   FailedCreate   statefulset/moco-ngw           create Claim mysql-data-moco-ngw-0 for Pod moco-ngw-0 in StatefulSet moco-ngw failed error: persistentvolumeclaims "mysql-data-moco-ngw-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>
2s          Warning   FailedCreate   statefulset/moco-ngw           create Pod moco-ngw-0 in StatefulSet moco-ngw failed error: failed to create PVC mysql-data-moco-ngw-0: persistentvolumeclaims "mysql-data-moco-ngw-0" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>

Environments

• Version: OpenShift 4.9
• OS: CoreOS

To Reproduce
Deploy the minimal cluster resource on the openshift cluster.

Expected behavior
PVC are created correctly

Additional context
This is a security enhancement in OpenShift, and is discussed in this BugZilla.
This could be resolved by extending the ClusterRole resources with additional /finalizers, but I need some help with this.

[ymmt2005]

Thank you for the report. We also do not have experience with OpenShift, so need help.

[zoetrope]

@cyberox
We do not have an OpenShift environment.
So could you try the following configuration in your environment?

kubectl edit clusterrole moco-manager-role

Please add the following rule in moco-manager-role:

- apiGroups:
  - ""
  resources:
  - persistentvolumeclaims/finalizers
  verbs:
  - update

If it works, we will add this configuration to our helm chart.

[cyberox]

Thank you for the suggestion. I applied it to the clusterrole, and redeployed the MySQLCluster resource.
I'm receiving the same error when creating the statefulset.

[zoetrope]

Please add the following rule as well.

- apiGroups:
  - apps
  resources:
  - statefulsets/finalizer
  verbs:
  - update

[cyberox]

I also added that to the clusterrole, but unfortunately it is not working.

[zoetrope]

I will try OpenShift free trial or Minishift.
Please wait a while.

[rassie]

So I've stumbled over this problem on my OKD 4.18 cluster. The reason this is happening is that OwnerReferencesPermissionEnforcement admission plugin is enabled by default in OpenShift/OKD, so this could happen in a pure Kubernetes cluster too depending on security pre-requisites.

After a long debugging session I've managed to come by with the following RBAC:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
    name: mysqlclusters-finalizer
rules:
    - apiGroups: ["moco.cybozu.com"]
      resources: ["mysqlclusters/finalizers"]
      verbs: ["update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
    name: mysqlclusters-finalizer-role-binding
subjects:
    - kind: ServiceAccount
      name: statefulset-controller
      namespace: kube-system
roleRef:
    kind: ClusterRole
    name: mysqlclusters-finalizer
    apiGroup: rbac.authorization.k8s.io

However, I want to stress that I don't think this is the proper way, giving out roles to system processes just feels wrong. I'd like to research how other operators handle this before proposing a change in Moco, but I'm not too optimistic I'd find much, because OpenShift support is too often an afterthought.

A connected issue is #777 -- OpenShift does not like hard-coded UIDs and GIDs and currently requires adding anyuid SCC to the service account to run Moco. Would be nice if that wasn't necessary.

@zoetrope @ymmt2005 could you re-open this one please? I think we might find a proper solution soon.

[ymmt2005]

Thank you for your report, @rassie !

[rassie]

I think I have identified the problem, at least in comparison with other operators. I think the owner reference on the PVC is wrong:

moco/controllers/mysqlcluster_controller.go

Lines 1963 to 1969 in 699d878

	pvc.WithOwnerReferences(metav1ac.OwnerReference().
	WithAPIVersion(apiVersion).
	WithKind(gvk.Kind).
	WithName(cluster.Name).
	WithUID(cluster.GetUID()).
	WithBlockOwnerDeletion(true).
	WithController(true))

From what I've seen from other operators, if a PVC has an owner reference, it's most commonly owned by the StatefulSet, not the custom resource which created the StatefulSet. Moco however sets MySQLCluster as PVC's ownerRef, so that when the statefulset-controller tries to create a PVC and set the blockOwnerDeletion option, it requires a permission to update finalizers on PVC's owner, which is MySQLCluster and not StatefulSet. statefulset-controller however, only has the permission for the latter, which is why everything fails.

I'm not quite sure why the owner reference is set like that, but I suspect it has something to do with PVC resizing support which needs a certain decoupling of StatefulSets and PVCs. I'm also not sure what best practices should be applied here, but I think the root cause is correct, but I can offer no full solution apart from not setting the ownerRef on the PVCs. YMMV as always.