DNS Issue - Nide can't resolve host

[tina-junold]

Description

I've running Homepage in my TrueNAS Scale. For some reasons Homepage can't resolve some hosts. I was working before, but I'm unsure if it is because i use OpenWRT for some days ore another reason. On this OpenWRT Adguard Home is the main DNS (on port 53 and others) and queries OpenWRTs dnsmasq (on port 54) to resolve the hostsnames.

Adguard shows me in the log nodejs on the TrueNAS queries the DNS

Einzelheiten der Antwort
Status
Verarbeitet
DNS-Server
10.10.10.254:54
Verstrichen
1 ms
Antwortcode
NOERROR
Antwort
A: 10.10.10.121 (ttl=0) 

so I'm out of ideas whats wrong

homepage version

v0.10.9

Installation method

Docker

Configuration

services.yaml

    - ArgoCD:
        href: https://argocd.infra.internal
        icon: si-argo
        widget:
          type: argocd
          url: https://argocd.infra.internal
          key: XYZ

    - Traefik (Services):
        href: https://traefik.infra.internal
        icon: si-traefikproxy
        widget:
          type: traefik
          url: https://traefik.svc.internal
          username: traefik
          password: XZY
    - Traefik (Infrastructure):
        href: https://traefik.infra.internal
        icon: si-traefikproxy
        widget:
          type: traefik
          url: https://traefik.infra.internal
          username: traefik
          password: XYZ

Container Logs

[2025-02-22T05:26:43.444Z] error: <httpProxy> Error calling https://argocd.infra.internal/api/v1/applications...
[2025-02-22T05:26:43.445Z] error: <httpProxy> [
  500,
  Error: getaddrinfo ENOTFOUND argocd.infra.internal
      at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'argocd.infra.internal'
  }
]
[2025-02-22T05:26:43.445Z] error: <credentialedProxyHandler> HTTP Error 500 calling https://argocd.infra.internal/api/v1/applications
[2025-02-22T05:26:43.446Z] error: <httpProxy> Error calling https://traefik.infra.internal/api/overview...
[2025-02-22T05:26:43.446Z] error: <httpProxy> [
  500,
  Error: getaddrinfo ENOTFOUND traefik.infra.internal
      at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'traefik.infra.internal'
  }
]
[2025-02-22T05:26:43.447Z] error: <httpProxy> Error calling https://traefik.svc.internal/api/overview...
[2025-02-22T05:26:43.447Z] error: <httpProxy> [
  500,
  Error: getaddrinfo ENOTFOUND traefik.svc.internal
      at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'traefik.svc.internal'
  }
]
``

### Browser Logs

_No response_

### Troubleshooting

First, i run dig on TrueNAS to ensure i can resolve the hostnames

dig traefik.infra.internal @10.10.10.254

; <<>> DiG 9.18.24-1-Debian <<>> traefik.infra.internal @10.10.10.254
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31322
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;traefik.infra.internal. IN A

;; ANSWER SECTION:
traefik.infra.internal. 0 IN A 10.10.10.120

;; Query time: 0 msec
;; SERVER: 10.10.10.254#53(10.10.10.254) (UDP)
;; WHEN: Sat Feb 22 17:05:53 CET 2025
;; MSG SIZE rcvd: 67

dig traefik.svc.internal @10.10.10.254

; <<>> DiG 9.18.24-1-Debian <<>> traefik.svc.internal @10.10.10.254
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45053
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;traefik.svc.internal. IN A

;; ANSWER SECTION:
traefik.svc.internal. 0 IN A 10.10.10.121

;; Query time: 4 msec
;; SERVER: 10.10.10.254#53(10.10.10.254) (UDP)
;; WHEN: Sat Feb 22 17:06:14 CET 2025

Than, i checked the daemon.json on TrueNAS. I also manually forced `dns`, `dns-search` and `dns-opts` later.

```json
{"data-root": "/mnt/.ix-apps/docker", "exec-opts": ["native.cgroupdriver=cgroupfs"], "iptables": true, "storage-driver": "overlay2", "default-address-pools": [{"base": "172.17.0.0/12", "size": 24}], "dns": ["10.10.10.254"], "dns-search": ["internal"], "dns-opts": ["ndots:1"]}

And ensure that the resolv.conf inside the container is correct.
resolv.conf inside docker container

# docker exec -it ix-homepage-homepage-1 sh
/app $ cat /etc/resolv.conf 
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 10.10.10.254
search internal
options ndots:1

# Based on host file: '/etc/resolv.conf'
# Overrides: [nameservers search options]
# Option ndots from: override

Than, i test lookup inside the homepage docker container calling node directly

$ node -pe 'require("dns").lookup("traefik.svc.internal",function(){console.dir(arguments)})'
GetAddrInfoReqWrap {
  callback: [Function (anonymous)],
  family: 0,
  hostname: 'traefik.svc.internal',
  oncomplete: [Function: onlookup]
}
[Arguments] {
  '0': Error: getaddrinfo ENOTFOUND traefik.svc.internal
      at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'traefik.svc.internal'
  }
}

[shamoon]

This seems like a docker networking question more than a homepage one. Have a look at https://docs.docker.com/engine/network/ and maybe ask over in a docker-specific forum, or a truenas one. If the hostname isn’t resolving then the fix will have nothing to do with homepage

[tina-junold]

@shamoon I don't think so. I think it's an container issue. I created a "custom app" in TrueNAS with my debug container that i'm using a lot on work if something is broken. It got the same network settings from docker like homepage. Since the homepage container does not contain, this is the only option to ensure dns is working as expected. As you can see below, the IP can resolved correctly.

docker exec ix-debug-debug-1 bash
root@truenas[~]# dig traefik.infra.internal

; <<>> DiG 9.18.24-1-Debian <<>> traefik.infra.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64805
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;traefik.infra.internal.		IN	A

;; ANSWER SECTION:
traefik.infra.internal.	0	IN	A	10.10.10.120

;; Query time: 0 msec
;; SERVER: 10.10.10.254#53(10.10.10.254) (UDP)
;; WHEN: Sun Feb 23 07:21:21 CET 2025
;; MSG SIZE  rcvd: 67
``

[shamoon]

I mean, our image is based on a vanilla node-alpine base image and as you noted, dns resolution doesn't work directly with node either. We're not doing anything special (or really, anything) with dns, so I don't see how.

[tina-junold]

Mmh, I might be dive deeper into the node container, but I'm not sure if i want to ;-)

BTW: I saw that you also work on paperless-ngx, so really, really thank you for your work!!!! I migrated almost every document (and there where a lot) to paperless and I don't know what I should do without it.

[shamoon]

Ha, the FOSS world is small, thanks ❤️

Yea I’d be happy to help if there’s some way, but I’m not really sure where the issue is, but let us know if you do figure something out

[shamoon]

Ps. I’m not sure if aliases work in your situation but maybe https://docs.docker.com/reference/compose-file/services/#aliases

[tina-junold]

I think this will not work. Beside the TrueNAS Server I host several VMs in my Proxmox cluster including multiple Kubernetes Clusters. These clusters. Are not part of the docker network.

[tina-junold]

node:alpine

/ # node -pe 'require("dns").lookup("traefik.svc.internal",function(){console.dir(arguments)})'
[Arguments] {
  '0': Error: getaddrinfo ENOTFOUND traefik.svc.internal
      at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'traefik.svc.internal'
  }
}
GetAddrInfoReqWrap {
  callback: [Function (anonymous)],
  family: 0,
  hostname: 'traefik.svc.internal',
  oncomplete: [Function: onlookup]
}

node:slim

# node -pe 'require("dns").lookup("traefik.svc.internal",function(){console.dir(arguments)})'
[Arguments] { '0': null, '1': '10.10.10.121', '2': 4 }
GetAddrInfoReqWrap {
  callback: [Function (anonymous)],
  family: 0,
  hostname: 'traefik.svc.internal',
  oncomplete: [Function: onlookup]
}

I found this: https://bell-sw.com/blog/how-to-deal-with-alpine-dns-issues/ . Now i try to find the packages which may help alpine to work correctly with DNS (maybe gcompat)

[tina-junold]

how strange would it be to request a homepage slim (in addition to alpine) container :-D

[shamoon]

Wow thanks for the deep dive, interesting indeed.

Unfortunately, I don’t think we’ll switch just for fear of breaking things and no we’re not going to build two versions just for this (we are gifted the compute for our CI from DigitalOcean).

I guess the question is whether there is a less-disruptive way to make it work (which is why I suggested the aliases thing). Maybe running a small container for dns resolution?

[tina-junold]

Some more findings:

Query with specific ip family

/ # node -pe 'require("dns").lookup("traefik.svc.internal",{family: 4},function(){console.dir(arguments)})'
[Arguments] { '0': null, '1': '10.10.10.121', '2': 4 }
GetAddrInfoReqWrap {
  callback: [Function (anonymous)],
  family: 4,
  hostname: 'traefik.svc.internal',
  oncomplete: [Function: onlookup]
}

/ # node -pe 'require("dns").lookup("traefik.svc.internal",{family: 6},function(){console.dir(arguments)})'
[Arguments] {
  '0': Error: getaddrinfo ENOTFOUND traefik.svc.internal
      at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:109:26) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'traefik.svc.internal'
  }

As you can see, the issue seems to be that the default setting in node is to use IPv6 to query dns server.

Therefore I dry to dig in the a node debug container which gives me:

# dig -6 traefik.svc.internal
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused
;; communications error to ::1#53: connection refused

; <<>> DiG 9.18.34 <<>> -6 traefik.svc.internal
;; global options: +cmd
;; no servers could be reached

# dig -6 traefik.svc.internal @fd4f:96f5:147a::1
;; UDP setup with fd4f:96f5:147a::1#53(fd4f:96f5:147a::1) for traefik.svc.internal failed: network unreachable.
;; no servers could be reached
;; UDP setup with fd4f:96f5:147a::1#53(fd4f:96f5:147a::1) for traefik.svc.internal failed: network unreachable.
;; no servers could be reached
;; UDP setup with fd4f:96f5:147a::1#53(fd4f:96f5:147a::1) for traefik.svc.internal failed: network unreachable.
;; no servers could be reached

Doing the same on the TrueNAS which is working

dig -6 traefik.svc.internal @fd4f:96f5:147a::1

; <<>> DiG 9.18.24-1-Debian <<>> -6 traefik.svc.internal @fd4f:96f5:147a::1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34098
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;traefik.svc.internal.		IN	A

;; ANSWER SECTION:
traefik.svc.internal.	0	IN	A	10.10.10.121

;; Query time: 0 msec
;; SERVER: fd4f:96f5:147a::1#53(fd4f:96f5:147a::1) (UDP)
;; WHEN: Mon Feb 24 01:16:50 CET 2025
;; MSG SIZE  rcvd: 65

I thought setup the IPv6 address as DNS in docker would resolve the issue, but it doesn't. based on the dig above he tries to resolve via ::1 instead of fd4f:96f5:147a::1.

[shamoon]

Curious if the dev image behaves any differently

[tina-junold]

Which dev image?

[shamoon]

Tagged dev