Investigate random DNS failures

[cohoe]

The resolver sporadically fails to resolve names. Seems random and not repeatable. Could be a cache and/or rate limit thing?

Also could be v6 related:

[1624389218] libunbound[21:0] error: udp connect failed: Cannot assign requested address for 2001:500:12::d0d port 53
[1624389218] libunbound[21:0] error: udp connect failed: Cannot assign requested address for 2001:503:ba3e::2:30 port 53
[1624389218] libunbound[21:0] error: udp connect failed: Cannot assign requested address for 2001:503:c27::2:30 port 53
[1624389218] libunbound[21:0] error: udp connect failed: Cannot assign requested address for 2001:500:2d::d port 53
[1624389218] libunbound[21:0] error: udp connect failed: Cannot assign requested address for 2001:dc3::35 port 53

[cohoe]

In all my kickstarting and configuring adventures I have not yet seen this error so I think the fix above may have done the trick. Will re-open if we see it at camp.

[cohoe]

Seeing more of this around middle weekend and start of second week.

[cohoe]

lancache dns has a single upstream of 8.8.8.8 which occasionally is returning servfails. unbound is returning the result from named.

[cohoe]

/etc/bind/named.conf.options

        forwarders {
                8.8.8.8;
                8.8.4.4;
                1.1.1.1;
        }

Manually added this in 2023 but it's still failing in 2024.

Added the 192.168.0.1 resolver and that seemed to work better

[cohoe]

/var/log/named/ mount local