patches: patch support e2e are still broken

[neolit123]

we made the e2e tests use the v1beta3.Init|JoinConfiguration.patches.directory feature:
#2545
#2537

but the CI is failing:

https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-patches-latest
https://storage.googleapis.com/kubernetes-jenkins/logs/ci-kubernetes-e2e-kubeadm-kinder-patches-latest/1427291038157901824/build-log.txt

looks like the patches are being copied to the nodes:

time="15:35:27" level=debug msg="Running: [docker cp /tmp/kubeadm-patches/kube-scheduler0+strategic.yaml kinder-patches-control-plane-1:/tmp/kubeadm-patches/kube-scheduler0+strategic.yaml]"
Error: No such container:path: kinder-patches-control-plane-1:/tmp/kubeadm-patches

the kubeadm config file contains the patches.directory path:

time="15:36:47" level=debug msg="generated config:\napiVersion: kubeadm.k8s.io/v1beta3\ncontrolPlane:\n localAPIEndpoint:\n advertiseAddress: 172.17.0.4\n bindPort: 6443\ndiscovery:\n bootstrapToken:\n apiServerEndpoint: 172.17.0.5:6443\n token: abcdef.0123456789abcdef\n unsafeSkipCAVerification: true\nkind: JoinConfiguration\nmetadata:\n name: config\nnodeRegistration:\n criSocket: /var/run/dockershim.sock\n kubeletExtraArgs:\n node-ip: 172.17.0.4\npatches:\n directory: /tmp/kubeadm-patches\n"
time="15:36:47" level=debug msg="Running: [docker cp /tmp/kinder-patches-control-plane-2-283719042 kinder-patches-control-plane-2:/kind/kubeadm.conf]"
time="15:36:47" level=debug msg="Running: [docker exec kinder-patches-control-plane-2 kubeadm version -o=short]"

but then kubeadm init finds no patches in the given directory /tmp/kubeadm-patches:

[patches] Reading patches from path "/tmp/kubeadm-patches"
...no patches were found here.

so either the patches were not copied correctly, or there is a bug in kubeadm.
will try to investigate locally.

[neolit123]

cc @pacoxu

[neolit123]

looks like docker cp somepatch container:path is failing.

kinder-patches-control-plane-1:$ Importing patches from /tmp/kubeadm-patches
time="19:50:17" level=debug msg="Running: [docker exec kinder-patches-control-plane-1 mkdir -p /tmp/kubeadm-patches]"
etcd+json.json
time="19:50:17" level=debug msg="Running: [docker cp /tmp/kubeadm-patches/etcd+json.json kinder-patches-control-plane-1:/tmp/kubeadm-patches/etcd+json.json]"
Error: No such container:path: kinder-patches-control-plane-1:/tmp/kubeadm-patches
kube-apiserver.yaml
time="19:50:17" level=debug msg="Running: [docker cp /tmp/kubeadm-patches/kube-apiserver.yaml kinder-patches-control-plane-1:/tmp/kubeadm-patches/kube-apiserver.yaml]"
Error: No such container:path: kinder-patches-control-plane-1:/tmp/kubeadm-patches
kube-controller-manager+merge.yaml
time="19:50:17" level=debug msg="Running: [docker cp /tmp/kubeadm-patches/kube-controller-manager+merge.yaml kinder-patches-control-plane-1:/tmp/kubeadm-patches/kube-controller-manager+merge.yaml]"
Error: No such container:path: kinder-patches-control-plane-1:/tmp/kubeadm-patches
kube-scheduler0+strategic.yaml
time="19:50:17" level=debug msg="Running: [docker cp /tmp/kubeadm-patches/kube-scheduler0+strategic.yaml kinder-patches-control-plane-1:/tmp/kubeadm-patches/kube-scheduler0+strategic.yaml]"
Error: No such container:path: kinder-patches-control-plane-1:/tmp/kubeadm-patches

this is caused by this change:
https://github.com/kubernetes/kubeadm/pull/2545/files#diff-f277f4b0d68900b4ebba057a2753bed34b77b7ece1db8e11c0619ef9ceb92328R116

i have no explanation why this is happening but i can reproduce it locally.

docker cp /tmp/kubeadm-patches/kube-apiserver.yaml kinder-patches-control-plane-1:/tmp/kube-apiserver.yaml

the command passes, the file is not copied under tmp.

docker cp /tmp/kubeadm-patches/kube-apiserver.yaml kinder-patches-control-plane-1:/tmp/kubeadm-patches/kube-apiserver.yaml

it either has this error:

Error response from daemon: extraction point is not a directory

or

Error: No such container:path: kinder-patches-control-plane-1:/tmp/kubeadm-patches

but the path /tmp/kubeadm-patches/ is a valid directory.

i will send a PR to revert:
https://github.com/kubernetes/kubeadm/pull/2545/files#diff-f277f4b0d68900b4ebba057a2753bed34b77b7ece1db8e11c0619ef9ceb92328R116

[neolit123]

last run is green:
https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kubeadm#kubeadm-kinder-patches-latest