Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CentOS 8 / RHEL 8: FirewallD + internal Network: Container <-> Container communication impossible: "Host is unreachable" #1353

Closed
2 of 3 tasks
Schuwi opened this issue Jan 30, 2022 · 1 comment
Closed
2 of 3 tasks

CentOS 8 / RHEL 8: FirewallD + internal Network: Container <-> Container communication impossible: "Host is unreachable" #1353

Schuwi opened this issue Jan 30, 2022 · 1 comment

Comments

@Schuwi
Copy link

Schuwi commented Jan 30, 2022 •
edited
Loading

  • This is a bug report
  • This is a feature request
  • I searched existing issues before opening this one

Expected behavior

Two containers connected to the same internal docker network should be able to communicate to each other.

Actual behavior

Pinging one container from inside another works fine but establishing a TCP connection (in this case connecting to an nginx server) fails with a "Host is unreachable" error (or in case of RHEL 8 on AWS a "No route to host" error).

Steps to reproduce the behavior

  • Create an AWS EC2 instance "Red Hat Enterprise Linux 8 (HVM), SSD Volume Type - ami-06ec8443c2a35b0ba (64-bit x86)"

  • Install firewalld

    $ sudo yum install firewalld
$ sudo systemctl start firewalld

  • Install docker

    $ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
$ sudo yum install docker-ce docker-ce-cli containerd.io
$ sudo systemctl start docker

  • Create internal network and two containers

    # docker network create --internal internal_net
# docker run --name nignx --network internal_net -d nginx
# docker run --name client --network internal_net -dit busybox

  • Attach to client and try to connect to nginx
    As you can see pinging nginx works fine

    [root@ip-172-31-25-11 ~]# docker attach client
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
31: eth0@if32: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue
    link/ether 02:42:ac:13:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.19.0.3/16 brd 172.19.255.255 scope global eth0
       valid_lft forever preferred_lft forever
/ # ping 172.19.0.2
PING 172.19.0.2 (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.114 ms
64 bytes from 172.19.0.2: seq=1 ttl=64 time=0.100 ms
64 bytes from 172.19.0.2: seq=2 ttl=64 time=0.099 ms
64 bytes from 172.19.0.2: seq=3 ttl=64 time=0.099 ms
^C
--- 172.19.0.2 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.099/0.103/0.114 ms

    Opening a TCP connection to port 80 however doesn't work

    / # nc -v 172.19.0.2 80
nc: 172.19.0.2 (172.19.0.2:80): No route to host

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:22 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:44 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)

Server:
 Containers: 3
  Running: 1
  Paused: 0
  Stopped: 2
 Images: 2
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.18.0-305.el8.x86_64
 Operating System: Red Hat Enterprise Linux 8.4 (Ootpa)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 806.4MiB
 Name: ip-172-31-25-11.eu-central-1.compute.internal
 ID: 5CES:3ENN:PIOE:EOAR:E56H:WU2I:N76H:PO5J:FYZA:S3OO:V62Y:6XD3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)

I originally encountered this issue on CentOS 8 running on a physical server.

I also noticed that the bridge network device (in this case br-c81932d5907d) is not assigned to the docker zone in firewalld (or any zone for that matter). It uses whichever zone is set as default.

nftables trace (nft add chain bridge filter trace_chain { type filter hook forward priority -301\; }) while running nc -v 172.19.0.2:

trace id d91c35db bridge filter trace_chain packet: iif "veth89eeb77" oif "veth1cd76c2" ether saddr 02:42:ac:13:00:03 ether daddr ff:ff:ff:ff:ff:ff arp operation request
trace id d91c35db bridge filter trace_chain rule meta nftrace set 1 (verdict continue)
trace id d91c35db bridge filter trace_chain verdict continue
trace id d91c35db bridge filter trace_chain policy accept
trace id d91c35db bridge filter FORWARD verdict continue
trace id d91c35db bridge filter FORWARD policy accept
trace id d91c35db bridge nat POSTROUTING verdict continue
trace id d91c35db bridge nat POSTROUTING policy accept
trace id 13ca8ca9 bridge filter trace_chain packet: iif "veth1cd76c2" oif "veth89eeb77" ether saddr 02:42:ac:13:00:02 ether daddr 02:42:ac:13:00:03 arp operation reply
trace id 13ca8ca9 bridge filter trace_chain rule meta nftrace set 1 (verdict continue)
trace id 13ca8ca9 bridge filter trace_chain verdict continue
trace id 13ca8ca9 bridge filter trace_chain policy accept
trace id 13ca8ca9 bridge filter FORWARD verdict continue
trace id 13ca8ca9 bridge filter FORWARD policy accept
trace id 13ca8ca9 bridge nat POSTROUTING verdict continue
trace id 13ca8ca9 bridge nat POSTROUTING policy accept
trace id a0369369 bridge filter trace_chain packet: iif "veth89eeb77" oif "veth1cd76c2" ether saddr 02:42:ac:13:00:03 ether daddr 02:42:ac:13:00:02 ip saddr 172.19.0.3 ip daddr 172.19.0.2 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 11143 ip protocol tcp ip length 60 tcp sport 39771 tcp dport 80 tcp flags == syn tcp window 29200
trace id a0369369 bridge filter trace_chain rule meta nftrace set 1 (verdict continue)
trace id a0369369 bridge filter trace_chain verdict continue
trace id a0369369 bridge filter trace_chain policy accept
trace id a0369369 bridge filter FORWARD verdict continue
trace id a0369369 bridge filter FORWARD policy accept
trace id a0369369 ip mangle FORWARD verdict continue
trace id a0369369 ip mangle FORWARD policy accept
trace id a0369369 ip filter FORWARD packet: iif "br-c81932d5907d" oif "br-c81932d5907d" ether saddr 02:42:ac:13:00:03 ether daddr 02:42:ac:13:00:02 ip saddr 172.19.0.3 ip daddr 172.19.0.2 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 11143 ip length 60 tcp sport 39771 tcp dport 80 tcp flags == syn tcp window 29200
trace id a0369369 ip filter FORWARD rule counter packets 31 bytes 2290 jump DOCKER-USER (verdict jump DOCKER-USER)
trace id a0369369 ip filter DOCKER-USER verdict return
trace id a0369369 ip filter FORWARD rule counter packets 31 bytes 2290 jump DOCKER-ISOLATION-STAGE-1 (verdict jump DOCKER-ISOLATION-STAGE-1)
trace id a0369369 ip filter DOCKER-ISOLATION-STAGE-1 verdict return
trace id a0369369 ip filter FORWARD rule iifname "br-c81932d5907d" oifname "br-c81932d5907d" counter packets 27 bytes 1990 accept (verdict accept)
trace id a0369369 inet firewalld filter_FORWARD packet: iif "br-c81932d5907d" oif "br-c81932d5907d" ether saddr 02:42:ac:13:00:03 ether daddr 02:42:ac:13:00:02 ip saddr 172.19.0.3 ip daddr 172.19.0.2 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 11143 ip protocol tcp ip length 60 tcp sport 39771 tcp dport 80 tcp flags == syn tcp window 29200
trace id a0369369 inet firewalld filter_FORWARD rule jump filter_FORWARD_POLICIES_pre (verdict jump filter_FORWARD_POLICIES_pre)
trace id a0369369 inet firewalld filter_FORWARD_POLICIES_pre verdict continue
trace id a0369369 inet firewalld filter_FORWARD rule jump filter_FORWARD_IN_ZONES_SOURCE (verdict jump filter_FORWARD_IN_ZONES_SOURCE)
trace id a0369369 inet firewalld filter_FORWARD_IN_ZONES_SOURCE verdict continue
trace id a0369369 inet firewalld filter_FORWARD rule jump filter_FORWARD_IN_ZONES (verdict jump filter_FORWARD_IN_ZONES)
trace id a0369369 inet firewalld filter_FORWARD_IN_ZONES rule goto filter_FWDI_public (verdict goto filter_FWDI_public)
trace id a0369369 inet firewalld filter_FWDI_public rule jump filter_FWDI_public_pre (verdict jump filter_FWDI_public_pre)
trace id a0369369 inet firewalld filter_FWDI_public_pre verdict continue
trace id a0369369 inet firewalld filter_FWDI_public rule jump filter_FWDI_public_log (verdict jump filter_FWDI_public_log)
trace id a0369369 inet firewalld filter_FWDI_public_log verdict continue
trace id a0369369 inet firewalld filter_FWDI_public rule jump filter_FWDI_public_deny (verdict jump filter_FWDI_public_deny)
trace id a0369369 inet firewalld filter_FWDI_public_deny verdict continue
trace id a0369369 inet firewalld filter_FWDI_public rule jump filter_FWDI_public_allow (verdict jump filter_FWDI_public_allow)
trace id a0369369 inet firewalld filter_FWDI_public_allow verdict continue
trace id a0369369 inet firewalld filter_FWDI_public rule jump filter_FWDI_public_post (verdict jump filter_FWDI_public_post)
trace id a0369369 inet firewalld filter_FWDI_public_post verdict continue
trace id a0369369 inet firewalld filter_FWDI_public verdict continue
trace id a0369369 inet firewalld filter_FORWARD rule jump filter_FORWARD_OUT_ZONES_SOURCE (verdict jump filter_FORWARD_OUT_ZONES_SOURCE)
trace id a0369369 inet firewalld filter_FORWARD_OUT_ZONES_SOURCE verdict continue
trace id a0369369 inet firewalld filter_FORWARD rule jump filter_FORWARD_OUT_ZONES (verdict jump filter_FORWARD_OUT_ZONES)
trace id a0369369 inet firewalld filter_FORWARD_OUT_ZONES rule goto filter_FWDO_public (verdict goto filter_FWDO_public)
trace id a0369369 inet firewalld filter_FWDO_public rule jump filter_FWDO_public_pre (verdict jump filter_FWDO_public_pre)
trace id a0369369 inet firewalld filter_FWDO_public_pre verdict continue
trace id a0369369 inet firewalld filter_FWDO_public rule jump filter_FWDO_public_log (verdict jump filter_FWDO_public_log)
trace id a0369369 inet firewalld filter_FWDO_public_log verdict continue
trace id a0369369 inet firewalld filter_FWDO_public rule jump filter_FWDO_public_deny (verdict jump filter_FWDO_public_deny)
trace id a0369369 inet firewalld filter_FWDO_public_deny verdict continue
trace id a0369369 inet firewalld filter_FWDO_public rule jump filter_FWDO_public_allow (verdict jump filter_FWDO_public_allow)
trace id a0369369 inet firewalld filter_FWDO_public_allow verdict continue
trace id a0369369 inet firewalld filter_FWDO_public rule jump filter_FWDO_public_post (verdict jump filter_FWDO_public_post)
trace id a0369369 inet firewalld filter_FWDO_public_post verdict continue
trace id a0369369 inet firewalld filter_FWDO_public verdict continue
trace id a0369369 inet firewalld filter_FORWARD rule jump filter_FORWARD_POLICIES_post (verdict jump filter_FORWARD_POLICIES_post)
trace id a0369369 inet firewalld filter_FORWARD_POLICIES_post verdict continue
trace id a0369369 inet firewalld filter_FORWARD rule reject with icmpx type admin-prohibited (verdict drop)
trace id 6d9a826f bridge filter trace_chain packet: iif "veth89eeb77" oif "veth1cd76c2" ether saddr 02:42:ac:13:00:03 ether daddr 02:42:ac:13:00:02 ip saddr 172.19.0.3 ip daddr 172.19.0.2 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 11144 ip protocol tcp ip length 60 tcp sport 39771 tcp dport 80 tcp flags == syn tcp window 29200
trace id 6d9a826f bridge filter trace_chain rule meta nftrace set 1 (verdict continue)
trace id 6d9a826f bridge filter trace_chain verdict continue
trace id 6d9a826f bridge filter trace_chain policy accept
trace id 6d9a826f bridge filter FORWARD verdict continue
trace id 6d9a826f bridge filter FORWARD policy accept
trace id 6d9a826f ip mangle FORWARD verdict continue
trace id 6d9a826f ip mangle FORWARD policy accept
trace id 6d9a826f ip filter FORWARD packet: iif "br-c81932d5907d" oif "br-c81932d5907d" ether saddr 02:42:ac:13:00:03 ether daddr 02:42:ac:13:00:02 ip saddr 172.19.0.3 ip daddr 172.19.0.2 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 11144 ip length 60 tcp sport 39771 tcp dport 80 tcp flags == syn tcp window 29200
trace id 6d9a826f ip filter FORWARD rule counter packets 31 bytes 2290 jump DOCKER-USER (verdict jump DOCKER-USER)
trace id 6d9a826f ip filter DOCKER-USER verdict return
trace id 6d9a826f ip filter FORWARD rule counter packets 31 bytes 2290 jump DOCKER-ISOLATION-STAGE-1 (verdict jump DOCKER-ISOLATION-STAGE-1)
trace id 6d9a826f ip filter DOCKER-ISOLATION-STAGE-1 verdict return
trace id 6d9a826f ip filter FORWARD rule iifname "br-c81932d5907d" oifname "br-c81932d5907d" counter packets 27 bytes 1990 accept (verdict accept)
trace id 6d9a826f inet firewalld filter_FORWARD packet: iif "br-c81932d5907d" oif "br-c81932d5907d" ether saddr 02:42:ac:13:00:03 ether daddr 02:42:ac:13:00:02 ip saddr 172.19.0.3 ip daddr 172.19.0.2 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 11144 ip protocol tcp ip length 60 tcp sport 39771 tcp dport 80 tcp flags == syn tcp window 29200
trace id 6d9a826f inet firewalld filter_FORWARD rule jump filter_FORWARD_POLICIES_pre (verdict jump filter_FORWARD_POLICIES_pre)
trace id 6d9a826f inet firewalld filter_FORWARD_POLICIES_pre verdict continue
trace id 6d9a826f inet firewalld filter_FORWARD rule jump filter_FORWARD_IN_ZONES_SOURCE (verdict jump filter_FORWARD_IN_ZONES_SOURCE)
trace id 6d9a826f inet firewalld filter_FORWARD_IN_ZONES_SOURCE verdict continue
trace id 6d9a826f inet firewalld filter_FORWARD rule jump filter_FORWARD_IN_ZONES (verdict jump filter_FORWARD_IN_ZONES)
trace id 6d9a826f inet firewalld filter_FORWARD_IN_ZONES rule goto filter_FWDI_public (verdict goto filter_FWDI_public)
trace id 6d9a826f inet firewalld filter_FWDI_public rule jump filter_FWDI_public_pre (verdict jump filter_FWDI_public_pre)
trace id 6d9a826f inet firewalld filter_FWDI_public_pre verdict continue
trace id 6d9a826f inet firewalld filter_FWDI_public rule jump filter_FWDI_public_log (verdict jump filter_FWDI_public_log)
trace id 6d9a826f inet firewalld filter_FWDI_public_log verdict continue
trace id 6d9a826f inet firewalld filter_FWDI_public rule jump filter_FWDI_public_deny (verdict jump filter_FWDI_public_deny)
trace id 6d9a826f inet firewalld filter_FWDI_public_deny verdict continue
trace id 6d9a826f inet firewalld filter_FWDI_public rule jump filter_FWDI_public_allow (verdict jump filter_FWDI_public_allow)
trace id 6d9a826f inet firewalld filter_FWDI_public_allow verdict continue
trace id 6d9a826f inet firewalld filter_FWDI_public rule jump filter_FWDI_public_post (verdict jump filter_FWDI_public_post)
trace id 6d9a826f inet firewalld filter_FWDI_public_post verdict continue
trace id 6d9a826f inet firewalld filter_FWDI_public verdict continue
trace id 6d9a826f inet firewalld filter_FORWARD rule jump filter_FORWARD_OUT_ZONES_SOURCE (verdict jump filter_FORWARD_OUT_ZONES_SOURCE)
trace id 6d9a826f inet firewalld filter_FORWARD_OUT_ZONES_SOURCE verdict continue
trace id 6d9a826f inet firewalld filter_FORWARD rule jump filter_FORWARD_OUT_ZONES (verdict jump filter_FORWARD_OUT_ZONES)
trace id 6d9a826f inet firewalld filter_FORWARD_OUT_ZONES rule goto filter_FWDO_public (verdict goto filter_FWDO_public)
trace id 6d9a826f inet firewalld filter_FWDO_public rule jump filter_FWDO_public_pre (verdict jump filter_FWDO_public_pre)
trace id 6d9a826f inet firewalld filter_FWDO_public_pre verdict continue
trace id 6d9a826f inet firewalld filter_FWDO_public rule jump filter_FWDO_public_log (verdict jump filter_FWDO_public_log)
trace id 6d9a826f inet firewalld filter_FWDO_public_log verdict continue
trace id 6d9a826f inet firewalld filter_FWDO_public rule jump filter_FWDO_public_deny (verdict jump filter_FWDO_public_deny)
trace id 6d9a826f inet firewalld filter_FWDO_public_deny verdict continue
trace id 6d9a826f inet firewalld filter_FWDO_public rule jump filter_FWDO_public_allow (verdict jump filter_FWDO_public_allow)
trace id 6d9a826f inet firewalld filter_FWDO_public_allow verdict continue
trace id 6d9a826f inet firewalld filter_FWDO_public rule jump filter_FWDO_public_post (verdict jump filter_FWDO_public_post)
trace id 6d9a826f inet firewalld filter_FWDO_public_post verdict continue
trace id 6d9a826f inet firewalld filter_FWDO_public verdict continue
trace id 6d9a826f inet firewalld filter_FORWARD rule jump filter_FORWARD_POLICIES_post (verdict jump filter_FORWARD_POLICIES_post)
trace id 6d9a826f inet firewalld filter_FORWARD_POLICIES_post verdict continue
trace id 6d9a826f inet firewalld filter_FORWARD rule reject with icmpx type admin-prohibited (verdict drop)
@Schuwi Schuwi mentioned this issue Jan 30, 2022
Open
@Schuwi
Copy link
Author

Schuwi commented Jan 30, 2022

Closing this issue in favor of moby/libnetwork#2647 which seems to be the root cause.

@Schuwi Schuwi closed this as completed Jan 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Schuwi