com.docker.network.host_ipv4 does not working when multiple IPs

[n454149301]

My English is very poor, I use machine translation, please forgive me if there is any misunderstanding.

My server uses one network port, and the switch in front of the network port has two routers.
The addresses are:

1. 192.168.11.1 (default gateway)
2. 192.168.88.127 (second gateway)

I set up two static IPs, corresponding to two gateways respectively.

1. 192.168.11.115
2. 192.168.88.115

My requirement is that some mirrors use 192.168.11.1 as the gateway, and some mirrors use 192.168.88.127 as the gateway.

First is the netplan configuration of my ubuntu server.

# /etc/netplan/xxx.yaml

network:
    version: 2
    renderer: networkd
    ethernets:
        enp11s0:
            dhcp4: no
            dhcp6: no
            addresses: 
              - 192.168.11.115/24
              - 192.168.88.115/24
            routes:
              - to: default
                via: 192.168.11.1
                metric: 11
                table: 11
              - to: default
                via: 192.168.88.127
                metric: 88
                table: 88
            routing-policy:
              - from: 192.168.11.0/24
                table: 11
              - from: 192.168.88.0/24
                table: 88
            nameservers:
                addresses: [ 8.8.8.8 ]

A default route was manually added.

# sudo ip route add default via 192.168.11.1 dev enp11s0 onlink

# route -n

0.0.0.0         192.168.11.1    0.0.0.0         UG    0      0        0 enp11s0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-8e24dec57030
172.19.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-5aa6796964ac
172.22.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-b0a7cef8c13f
172.23.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-8cea7d07a39b
172.27.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-5e2fa258e919
172.31.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker_gwbridge
192.168.11.0    0.0.0.0         255.255.255.0   U     0      0        0 enp11s0
192.168.88.0    0.0.0.0         255.255.255.0   U     0      0        0 enp11s0

When I use the following command, I get the public network ip of the default gateway.

# curl https://cip.cc

IP	: xxx.xxx.0.76

When I use the following command, I get the public network ip of the second gateway.

# curl --interface 192.168.88.115 http://cip.cc

IP	:xxx.xxx.252.105

So far it has met my expectations. Next use the docker container.

First create the network.

# sudo docker network create -d bridge --opt com.docker.network.host_ipv4=192.168.88.115 my_test

# sudo docker network inspect my_test
[
    {
        "Name": "my_test",
        "Id": "20162b6a99db3c1b1fa44e244b4d2cfd6de20beeb6cb3f7b5ca9544c3c8cfaf8",
        "Created": "2022-06-20T01:22:45.556044108+08:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.24.0.0/16",
                    "Gateway": "172.24.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "9c86f5050247894794c384c8788df5a56fba50df59823a779ed5298f2dcafcf2": {
                "Name": "jolly_ptolemy",
                "EndpointID": "9eb5559d1d394c432f24d95f44f551b11d8dec136ffc364732ed256b586fe478",
                "MacAddress": "xx:xx:xx:18:00:02",
                "IPv4Address": "172.24.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.host_ipv4": "192.168.88.115"
        },
        "Labels": {}
    }
]

Then start the mirror.
The image I use is based on kalilinux/kali-last-release, added some tools, no other changes.

sudo docker run -itd --network my_test xx/kali-linux:v1

After entering the container, execute the curl command to view the public network ip.

# curl https://cip.cc

IP	: xxx.xxx.0.76

Because the --opt com.docker.network.host_ipv4=192.168.88.115 parameter exists.
The second gateway I am using now should get the public IP of xxx.xxx.252.105.

I tried to use tcpdump to capture the data packets of the network card of the physical machine, and compared the data packets of the same http request inside the container and outside the container, what is the difference.

# Packets received in the container

0000   48 0e ec 51 eb 39 d8 cb 8a fa 87 8d 08 00 45 00   H..Q.9........E.
0010   00 6e 3f b2 40 00 3f 06 c5 8f c0 a8 58 73 7a 33   .n?.@.?.....Xsz3
0020   a2 f9 9a ba 00 50 28 90 d9 6f 0c 74 8d 83 50 18   .....P(..o.t..P.
0030   01 f6 36 a9 00 00 47 45 54 20 2f 20 48 54 54 50   ..6...GET / HTTP
0040   2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 63 69 70 2e   /1.1..Host: cip.
0050   63 63 0d 0a 55 73 65 72 2d 41 67 65 6e 74 3a 20   cc..User-Agent: 
0060   63 75 72 6c 2f 37 2e 37 39 2e 31 0d 0a 41 63 63   curl/7.79.1..Acc
0070   65 70 74 3a 20 2a 2f 2a 0d 0a 0d 0a               ept: */*....

# Packets received outside the container
0000   00 e2 69 29 ed 18 d8 cb 8a fa 87 8d 08 00 45 00   ..i)..........E.
0010   00 7a d1 15 40 00 40 06 33 20 c0 a8 58 73 7a 33   .z..@[email protected] ..Xsz3
0020   a2 f9 b5 42 00 50 6a 18 1e d9 9a 69 d3 ea 80 18   ...B.Pj....i....
0030   01 f6 36 b5 00 00 01 01 08 0a 1f a8 a0 13 3b fa   ..6...........;.
0040   49 36 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31   I6GET / HTTP/1.1
0050   0d 0a 48 6f 73 74 3a 20 63 69 70 2e 63 63 0d 0a   ..Host: cip.cc..
0060   55 73 65 72 2d 41 67 65 6e 74 3a 20 63 75 72 6c   User-Agent: curl
0070   2f 37 2e 38 31 2e 30 0d 0a 41 63 63 65 70 74 3a   /7.81.0..Accept:
0080   20 2a 2f 2a 0d 0a 0d 0a                            */*....

Starting from 4500, two contents, the source IP is 192.168.88.115, and the destination IP is 122.51.162.249. The http body is all the same.

The source mac addresses are all d8:cb:8a:fa:87:8d.

It is successfully sent outside the container, and the IP of xxx.xxx.252.105 is obtained. The destination MAC is 00:e2:69:29:ed:18.
It is sent in the container, and the IP of the error xxx.xxx.0.76 is obtained, and the destination MAC is 48:0e:ec:51:eb:39.

How should I configure it to achieve my needs.

[n454149301]

docker version

Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:33 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:42 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0