Unifi OS 4.0.X - WireGuard VPN working (now with 100% less DNS leaks!)

[Unlearned6688]

Just wanted to post up a super quick "this worked for me" summary of fixes for future frustrated people.

Background summary:

Either due to changes in the 4.0.6 Unifi OS update (or possibly even before it), or something bugging out, etc. peacey's split-vpn script became inoperable on my machine. After "messing" around a bit, I got things back up and running by reinstalling the script. However, before I had totally eliminated all DNS leakage with WireGuard via peacey's script. This was no longer true! Thus, I set out to fix the DNS leaks.

Goals upon outset:

Get WireGuard split-vpn script running again with no DNS leaks detected. DNS leaks were not acceptable (to me) as "running"

My network setup:

-Commercial VPN from Mullvad and Surfshark (to try different things. both appear to work)
-UDM Pro with some ubiquiti switches, little hubs, APs, etc. that are irrelevant for this discussion.
-2x AdGuard DNS servers/filters on my "normal" usage VLAN (for my phones, laptop, etc.). 2 for redundancy
-multiple VLANs for network equipment such as UDM P, separate for devices, separate again for IoT, guest, etc.

My unifi utilities/split-vpn config:

(re)Followed all instructions to setup split-vpn including the optional portion that creates ipsets to filter based on domain. The idea being you can configure the internal/standard DNSmasq or another DNS server (pihole, adguard, etc.) to resolve domains as instructed and tunnel those IPs through a specific chosen VPN. Other domains/IPs would go out the WAN as normal.

As stated above, this worked (still unsure what had broken- but it started working upon reinstall so that's ok). However, per Surfshark AND Mullvad AND 3rd party sites/tools, the VPN was leaking aka not using the designated DNS for that VPN IP. I tried setting the DNS in the vpn.conf (inside the wireguard/[your-wg-provider]/) dir. It did absolutely nothing. The only thing the UDM P was responding to was changes inside /ipsets dir and then the VPN_domains.conf file(s). At the bottom, and peacey outlines this in his instructions, you can set a FORWARD_SERVER="x.x.x.x" to send all forced domains to that specific IP, local or external.

Problem: This did work, HOWEVER, Mullvad and Surfshark, both, again, did not respond well to this. ie, they both refuse to resolve when you force them to use their own, specific (from the wg0.conf file provided by them) DNS. Neither would work with that DNS, nor something like 1.1.1.1, nor my other local adguards, trust me when I assure you: I tried literally every DNS combo I could. Nothing worked as in it would still have DNS leakage (best case) or wouldn't resolve (worst).

Final changes that got everything working:

I finally went back to the basics because clearly it seems Ubiquiti has changed something about how their internal DNSmasq deals with things. They've been adding stuff like DNS forwards to their GUI (doesn't seem to fully work, btw, I tried that too of course). It's possible/probable that this has interfered with files, I don't know. I just wanted my VPN to work.

The basics being: several years back when you could easily install a pihole via podman (docker) on the host network of the UDM P. I installed AdGuard (being used for the same thing- purely as a DNS server) inside the nspawn container. It only takes ~15mins or so if you're somewhat quick-fingered. Install nspawn, set up a more isolated macvlan, install adguard, make sure it works, bam. pihole would work too most likely.

So, with the new adguard on a macvlan, I set it to be my FORWARD_SERVER in the ipsets VPN_domains.conf file. Then, under the wireguard settings dir, inside vpn.conf I set FORCED_SOURCE_IPV4= to my new adguard's IP. (interface set to brX X=VLAN would likely also work). This turned out to be the "winning" combo. Set the FORWARD_SERVER then force that server through the VPN. Then, everything that passes through it will also automatically get the VPN's DNS (Mullvad, etc. are SUPPOSED to sort of force their own DNS no matter what when using the VPN tunnel-> it seems the domains refusing to resolve was likely a consequence of an un-resolved conflict between the VPN server and the UDM Pro.)

The only other settings I have inside the vpn.conf file are (just so people know, but this is all basically "standard" per peacey's instructions):

FORCED_IPSETS="VPN_FORCED:dst VPN_FORCED:src"
EXEMPT_IPSETS="VPN_EXEMPT:dst"

DNS_IPV4_IP=
and
DNS_IPV4_INTERFACE=
(set to my new adguard IP and VLAN (brX). it will work without these being set, however, I left them set to hopefully "force" any devices that might try to ignore the ipsets stuff)

everything else is a normal setting or left unchanged

Quick Summary:

VPN was working, but DNS was leaking
Solution was using a new internal DNS server (adguard) and FORCING that DNS server through the VPN along with domains/IPs I wanted tunneled
VPN leakage was immediately resolved by forcing the new DNS server through the VPN.

Hopefully this is helpful for someone as we await ubiquiti staff fixing their buggy WG implementation...until then, thank you peacey for the script, thank you others for write-ups I probably read all of them, and I'm out!