Fix intermittent DNS timeouts on .lan search domain (breaks Sonarr/Prowlarr indexers) #47

New issue

Closed

opened 2026-08-08 10:23:39 +00:00 by lab · 1 comment

lab commented

2026-08-08 10:23:39 +00:00

Owner

Symptom

Sonarr indexer test fails:

Unable to connect to indexer: HTTP request failed: [429:TooManyRequests] [GET]
at [http://192.168.0.211:9696/2/api?t=tvsearch&cat=5000,5050,5040,5045&...]

Root cause

The 429 is not an upstream rate limit. Prowlarr answers it directly:

<error code="429" description="Indexer is disabled till 08/08/2026 22:57:33
due to recent failures." />

Prowlarr auto-disabled The Pirate Bay and 1337x after repeated
connection failures, logged as:

Cardigann: Unable to connect to indexer [https://apibay.org/...].
This is typically caused by DNS/SSL issues.

Why the indexers fail: intermittent DNS timeouts on `.lan` probes

Pod resolv.conf carries search ... lan with ndots:5.
Every external lookup (apibay.org, 1337x.to) first probes
*.svc.cluster.local, then *.lan, before the real name.
CoreDNS forwards .lan to 1.1.1.1 (node /etc/resolv.conf):
```
[ERROR] plugin/errors: 2 radarr.servarr.com.lan. A:
read udp ...->1.1.1.1:53: i/o timeout
```
361 .lan i/o timeouts in 96h, intermittent.
On timeout the .NET resolver returns EAGAIN: Resource temporarily unavailable — visible in Prowlarr logs for prowlarr.servarr.com:443,
indexers.prowlarr.com:443, apibay.org.
Measured: full search chain ~5s when .lan stalls vs 5ms direct query.
Prowlarr requests exceed their timeout, get marked failed, and after
repeated failures the indexer is auto-disabled for 24h.

DNS topology today (no Pi-hole)

AdGuard Home on the OpenWrt router at 192.168.0.1, Unbound for recursion.
AdGuard resolves external names fine (22ms) but SERVFAILs .lan
(13ms) — the .lan zone no longer exists on the network.
The stale lan search domain comes from DHCP/node resolv.conf and is
still inherited by pods.

Same DNS failure pattern: #43 (Authelia/LLDAP slow), #45 (Homarr OIDC
resolution failure).

Action plan

#	Fix	Detail
1	Remove stale `lan` search domain from pods	Configure kubelet resolv-conf so pods stop probing `*.lan`; the zone does not exist
2	Point CoreDNS forward at AdGuard `192.168.0.1`	Local recursive (Unbound) instead of `1.1.1.1`; fast, consistent, and serves DHCP records
3	Fix stale Prowlarr `Http` proxy	`qbittorrent-proxy.media.svc.cluster.local:8888` is NXDOMAIN; real service is `gluetun-proxy`
4	Verify	Re-test Sonarr indexers after disabled state clears; watch CoreDNS error rate

## Symptom Sonarr indexer test fails: ```text Unable to connect to indexer: HTTP request failed: [429:TooManyRequests] [GET] at [http://192.168.0.211:9696/2/api?t=tvsearch&cat=5000,5050,5040,5045&...] ``` ## Root cause The 429 is not an upstream rate limit. Prowlarr answers it directly: ```text <error code="429" description="Indexer is disabled till 08/08/2026 22:57:33 due to recent failures." /> ``` Prowlarr auto-disabled **The Pirate Bay** and **1337x** after repeated connection failures, logged as: ```text Cardigann: Unable to connect to indexer [https://apibay.org/...]. This is typically caused by DNS/SSL issues. ``` ### Why the indexers fail: intermittent DNS timeouts on `.lan` probes - Pod `resolv.conf` carries `search ... lan` with `ndots:5`. - Every external lookup (`apibay.org`, `1337x.to`) first probes `*.svc.cluster.local`, then `*.lan`, before the real name. - CoreDNS forwards `.lan` to `1.1.1.1` (node `/etc/resolv.conf`): ```text [ERROR] plugin/errors: 2 radarr.servarr.com.lan. A: read udp ...->1.1.1.1:53: i/o timeout ``` 361 `.lan` i/o timeouts in 96h, intermittent. - On timeout the .NET resolver returns `EAGAIN: Resource temporarily unavailable` — visible in Prowlarr logs for `prowlarr.servarr.com:443`, `indexers.prowlarr.com:443`, `apibay.org`. - Measured: full search chain ~5s when `.lan` stalls vs 5ms direct query. Prowlarr requests exceed their timeout, get marked failed, and after repeated failures the indexer is auto-disabled for 24h. ### DNS topology today (no Pi-hole) - AdGuard Home on the OpenWrt router at `192.168.0.1`, Unbound for recursion. - AdGuard resolves external names fine (22ms) but SERVFAILs `.lan` (13ms) — the `.lan` zone no longer exists on the network. - The stale `lan` search domain comes from DHCP/node resolv.conf and is still inherited by pods. ## Related Same DNS failure pattern: #43 (Authelia/LLDAP slow), #45 (Homarr OIDC resolution failure). ## Action plan | # | Fix | Detail | |---|-----|--------| | 1 | Remove stale `lan` search domain from pods | Configure kubelet resolv-conf so pods stop probing `*.lan`; the zone does not exist | | 2 | Point CoreDNS forward at AdGuard `192.168.0.1` | Local recursive (Unbound) instead of `1.1.1.1`; fast, consistent, and serves DHCP records | | 3 | Fix stale Prowlarr `Http` proxy | `qbittorrent-proxy.media.svc.cluster.local:8888` is NXDOMAIN; real service is `gluetun-proxy` | | 4 | Verify | Re-test Sonarr indexers after disabled state clears; watch CoreDNS error rate |

lab added the

bug

label

2026-08-08 10:23:46 +00:00

lab commented

2026-08-22 20:07:00 +00:00

Author

Owner

Fixed at root cause.

Root cause

/etc/resolv-no-search.conf on every K3s node contained
nameserver 127.0.0.53 (systemd-resolved stub). Kubelet passes this file to
pods via --resolv-conf, so CoreDNS's stock forward . /etc/resolv.conf
forwarded queries to itself -> plugin/loop fatal + intermittent timeouts,
breaking Sonarr/Prowlarr indexer lookups.

Fix

/etc/resolv-no-search.conf now points at 192.168.0.1 (AdGuard on the
router) on all four nodes, no search domains
CoreDNS ConfigMap left at stock forward-to-resolv.conf
Rolling-restarted nodes; verified pods resolve external names cleanly

Also masked multipathd.service + multipathd.socket on all nodes during
follow-up volume-mount debugging (separate root cause, see #49 comment).

Fixed at root cause. ## Root cause `/etc/resolv-no-search.conf` on every K3s node contained `nameserver 127.0.0.53` (systemd-resolved stub). Kubelet passes this file to pods via `--resolv-conf`, so CoreDNS's stock `forward . /etc/resolv.conf` forwarded queries to itself -> plugin/loop fatal + intermittent timeouts, breaking Sonarr/Prowlarr indexer lookups. ## Fix - `/etc/resolv-no-search.conf` now points at `192.168.0.1` (AdGuard on the router) on all four nodes, no search domains - CoreDNS ConfigMap left at stock forward-to-resolv.conf - Rolling-restarted nodes; verified pods resolve external names cleanly Also masked `multipathd.service` + `multipathd.socket` on all nodes during follow-up volume-mount debugging (separate root cause, see #49 comment).

lab closed this issue

2026-08-22 20:07:00 +00:00