Pull image shows dial tcp: lookup registry-1.docker.io on 172.19.0.1:53: no such host

[BarrieShieh]

What happened:
Hi,
I use kind to create a local cluster. My machine is behind the proxy, and when I start a deployment, the pod logs shows that can not dial tcp: lookup registry-1.docker.io 172.19.0.1:53: on no such host. I have already set the env in /etc/systemd/system/docker.service.d/http-proxy.conf and /etc/systemd/system/docker.service.d/https-proxy.conf. I used minikube to create the cluster, there was no such issue then.

Is there any special config to pass the proxy setting into the cluster?

Here is the log

 Events:                                                                                                                                      
   Type     Reason     Age                   From                            Message                                                          
   ----     ------     ----                  ----                            -------                                                          
   Normal   Scheduled  20m                   default-scheduler               Successfully assigned istio-system/istio-init-crd-all-1.5.6-z9jd 
 c to staging-control-plane                                                                                                                   
   Normal   Pulling    19m (x4 over 20m)     kubelet, staging-control-plane  Pulling image "docker.io/istio/kubectl:1.5.6"                    
   Warning  Failed     19m (x4 over 20m)     kubelet, staging-control-plane  Failed to pull image "docker.io/istio/kubectl:1.5.6": rpc error: 
  code = Unknown desc = failed to pull and unpack image "docker.io/istio/kubectl:1.5.6": failed to resolve reference "docker.io/istio/kubectl 
 :1.5.6": failed to do request: Head https://registry-1.docker.io/v2/istio/kubectl/manifests/1.5.6: dial tcp: lookup registry-1.docker.io on  
 172.19.0.1:53: no such host                                                                                                                  
   Warning  Failed     19m (x4 over 20m)     kubelet, staging-control-plane  Error: ErrImagePull                                              
   Normal   BackOff    5m35s (x65 over 20m)  kubelet, staging-control-plane  Back-off pulling image "docker.io/istio/kubectl:1.5.6"           
   Warning  Failed     30s (x87 over 20m)    kubelet, staging-control-plane  Error: ImagePullBackOff

What you expected to happen:
pull image success

How to reproduce it (as minimally and precisely as possible):
Deploy apps behind a proxy (docker daemon must also be correctly configured)

Environment:

• kind version: (use kind version): 0.9.0
• Kubernetes version: (use kubectl version): v1.19.1
• Docker version: (use docker info): 19.03.1
• OS (e.g. from /etc/os-release): Red Hat Enterprise Linux 7

[BenTheElder]

if you pass proxy environment variables to kind binary it will correctly pass them to the node and handle them, we have to inject things to no_proxy, however even when we've done that we wouldn't be injecting dockerhub to no_proxy, that's up to you?

[BenTheElder]

you've set proxy settings for the host docker by configuring those files, but we need proxy configuration for the kind nodes and their nested container runtime.

when you run the kind command it reads the proxy environment variables and respects those, configure the nodes to respect those, with additional endpoints internal to the cluster added to no_proxy.

these environment variables seem to be the closest thing to a standard for supporting proxies, it should work with them, but it is not running underneath the docker.service so just configuring that isn't really sufficient at the moment.

depending on how minikube was used the requirements will be different.

[BarrieShieh]

you've set proxy settings for the host docker by configuring those files, but we need proxy configuration for the kind nodes and their nested container runtime.

when you run the kind command it reads the proxy environment variables and respects those, configure the nodes to respect those, with additional endpoints internal to the cluster added to no_proxy.

these environment variables seem to be the closest thing to a standard for supporting proxies, it should work with them, but it is not running underneath the docker.service so just configuring that isn't really sufficient at the moment.

depending on how minikube was used the requirements will be different.

Hi,
I want to use the proxy for the nested container, not bypass the proxy. Sorry for the confusing. Because the server in my company cannot connect to the Internet directly. Every traffic has to go through the HTTP proxy. So as the traffic in the nested container. So the no_proxy might not helping. Is there any documentation of how to configure nested container also using HTTP proxy to pull images?

Thanks

[BenTheElder]

I want to use the proxy for the nested container, not bypass the proxy. Sorry for the confusing. Because the server in my company cannot connect to the Internet directly. Every traffic has to go through the HTTP proxy. So as the traffic in the nested container. So the no_proxy might not helping. Is there any documentation of how to configure nested container also using HTTP proxy to pull images?

You just need to set the environment variables when calling kind. I.E. in your shell.

So the no_proxy might not helping.

the no_proxy is not causing this, it's set to no_proxy traffic between the internal components of the cluster.

[BarrieShieh]

I want to use the proxy for the nested container, not bypass the proxy. Sorry for the confusing. Because the server in my company cannot connect to the Internet directly. Every traffic has to go through the HTTP proxy. So as the traffic in the nested container. So the no_proxy might not helping. Is there any documentation of how to configure nested container also using HTTP proxy to pull images?

You just need to set the environment variables when calling kind. I.E. in your shell.

So the no_proxy might not helping.

the no_proxy is not causing this, it's set to no_proxy traffic between the internal components of the cluster.

Thanks for the tips. I used command HTTP_PROXY=http://10.177.241.67:3128 HTTPS_PROXY=http://10.177.241.67:3128 kind create cluster --name staging --config ./kind_config. And it's working. But for the internal traffic, it will shows 404 error like blueprint-prometheus-opera-prometheus does not exist. I have to add no_proxy when I add a new services to the cluster, and add port mapping to the config manually. I don't use ingress, because there're some services like dev databases running in the cluster. I need to separated ports.

Is there anyway to add port mapping and no_proxy without recreating the whole cluster?

[BenTheElder]

Thanks for the tips. I used command HTTP_PROXY=http://10.177.241.67:3128 HTTPS_PROXY=http://10.177.241.67:3128 kind create cluster --name staging --config ./kind_config. And it's working

👍

But for the internal traffic, it will shows 404 error like blueprint-prometheus-opera-prometheus does not exist.

Internal to the cluster? We inject no_proxy for all the things we can predict (like the service namespace, node IP range, pod IP range, service VIP range), but we can't predict your pod / service names. If you use a more qualified name it should match (e.g. blueprint-prometheus-opera-prometheus.svc)

kind/pkg/cluster/internal/providers/docker/provision.go

Line 302 in c4dd799

noProxyList = append(noProxyList, ".svc", ".svc.cluster", ".svc.cluster.local")

I don't think we can reasonably do much more for proxies than we do now, we can't set no proxy for these arbitrary unqualified domains.

We could try to work with the proxy settings passed to dockerd (if they're exposed in some way) but generally we need the chance to read your no_proxy so we can inject all these additional ranges when creating the node, and it sounds like those were only setting the proxy, not no_proxy.

Is there anyway to add port mapping and no_proxy without recreating the whole cluster?

not really (docker restrictions). though on linux you don't need kind level port mapping, that's for portability to mac etc. on linux you can reach the nodes by IP.

recreating clusters is supposed to be cheap as well, but you could try to modify the http proxy configuration in systemd and restart all services on the node using docker exec.

... I really recommend recreating the cluster, it's not something binaries / dependencies we don't implement ourselves like containerd read dynamically, pretty universally proxy settings must be introduced before a process starts.

[BenTheElder]

I think anything remaining is at most a dupe of #1175