Managed routes overwrite existing routes on OPNsense

[darkain]

I have two OPNsense routers each with their own WAN connection and connected to the same shared LAN. Both are connecting to the same ZT network. Within that ZT network, one of the routers is set as a Managed Route gateway. The router that isnt the gateway has its LAN subnet pointing to the wrong interface and becomes entirely inaccessible from the LAN.

Local LAN: 192.168.100.0/24
ZT LAN: 192.168.200.0/24
Managed Route: 192.168.100.0/24 > 192.168.200.1

Router 1:
LAN: 192.168.100.1
ZT: 192.168.200.1

Router 2:
LAN: 192.168.100.5
ZT: 192.168.200.5

With this configuration, Router 2's LAN subnet (192.168.100.0/24) is now pointing to the ZT virtual address of Router 1 (192.168.200.1) instead of its own local interface, even though it is physically directly connected to that LAN.

Before adding a managed route locally, I think ZT should make the effort to see if that subnet already exists. If it does, no action should be taken.

[glimberg]

Sounds like in this case you want to set allowManaged=0 on router 2 and manage the IP and route manually. zerotier-cli set <network_id> allowManaged=0. Once set, ZeroTier won't automatically push the IP addresses or routes to that router.

[darkain]

If that is set, then ALL other managed routes disappear too. Also, using the CLI tool, will that setting persist between reboots, or is it session only?

[glimberg]

Yes, the allowManaged setting will persist between reboots, assuming you don't delete ZeroTier's data dir on reboot.

[laduke]

It'll stay, but you can also add it to a file in /var/lib/zerotier-one/networks.d/${networkdId}.local.conf, if that's easier to manage.

$ cat 1234565963d77bcd.local.conf 
allowManaged=1
allowGlobal=0
allowDefault=0

Other ideas:

• blacklist the address or interface in local.conf

• Change the ZT managed route to /23 - It'll be less specific than the LAN, so the os will use the LAN

[darkain]

I have several routes being managed, not just one. Using ZT as basically the central router between several building's LANs. Luckily, none of the LAN subnets are next to each other, so the /23 trick is working. But I still think that ZT shouldn't override/replace any routes that already exist on physical interfaces.

[jdrews]

Referring to #354
I think it would be better if allowManaged=0 would not set any managed routes, but still set the IP of the zt interface.

[darkain]

Curious if there has been any work on this bug at all? ZT should not be overwritting LAN interface routes/subnets. I was just bitten by this again on one of my networks. Luckily the other network is on OSPF now, and that is smart enough to know local vs remote routes and manage it properly.

[ihipop]

#750 (comment)
I think a router metric to static routers on my.zerotier.com is a better Solution

[reedog117]

I just got bitten by this on a complex CARP redundant gateway installation with OPNsense. The allowManaged setting doesn’t seem to be exposed in the OPNsense UI so I’m wondering if an update is needed to better manage this.

[ropeguru]

I am running into this same issue.

Two interfaces on my router, em0 and em1. Em0 is LAN and em1 is WAN.

LAN is 192.168.1.204
WAN is 10.1.10.15
ZT is 192.168.196.195
All subnets are /24

In ZeroTier I have a managed route for the 192.168.1.0/24 subnet so other location know how to get it from remote locations. When OpnSense boots, ZeroTier overwrites the connected interface entry in the routing table and at that point, neither the 192.168.1.204 nor the 192.168.196.195 addresses are reachable. As soon as the ZeroTier route is removed and the correct connected route entry is applied, both the LAN and ZT inferaces become available.

A connected interface's entry in a route table should never be overwritten.

Is this a ZeroTier issue or a OpnSense issue?

[webxI23]

Is this a ZeroTier issue or a OpnSense issue?

Got the same problems with connected routes here (ZT on Opnsense) and used the "OSPF over ZT" solution (or at least a working B-plan). I'm not seeing why it could be a Opnsense issue considering it's ZT that tries to inject new routes. About that, no way to compare the " ip route show " or wathever OS command before injecting the route (if it's already there, do nothing) ?

[ropeguru]

Yes, while it is ZT injecting the routes, what is OPNsense doing which might remove those roues after ZT puts them in. In my case, it seems that the routes initially get configured, but then at some point disappear.

I am also having an issue where the connection is going down, even with routes, as I am getting some kind of UDP flood between 2 internal interfaces, one on OPNsense and the other a ZT bridge interface, and it essentially becomes a DoS. Has only ever happened with OPNsense, and I am trying to track down the cause. It is almost like in OPNsense, ZT initially connectects using the WAN interface, that IP show up in the ZT console, but at some point it starts thinking that ZT's source IP is the LAN interface with the IP as the NAT address.. I need to see if there is a way to bind ZT to a specific interface.

[webxI23]

I need to see if there is a way to bind ZT to a specific interface.

Never had those bugs here but Zerotier cannot reach a root or leaf server via anything but WAN interface so binding is not really useful in this case.

Your other problems should probably be discussed on another post to not pollute the original topic.

[darkain]

I am also having an issue where the connection is going down, even with routes, as I am getting some kind of UDP flood between 2 internal interfaces, one on OPNsense and the other a ZT bridge interface, and it essentially becomes a DoS. Has only ever happened with OPNsense, and I am trying to track down the cause. It is almost like in OPNsense, ZT initially connectects using the WAN interface, that IP show up in the ZT console, but at some point it starts thinking that ZT's source IP is the LAN interface with the IP as the NAT address.. I need to see if there is a way to bind ZT to a specific interface.

This particular issue is well known and documented at this point, see this other open ticket: #779

[ropeguru]

This particular issue is well known and documented at this point, see this other open ticket: #779

Thanks.. I tried searching but apparently the terms I was using were just not adequate enough..