fix(infra): investigate k8s-worker-01 node failure and stuck pods

[lexfrei]

Summary

Worker node k8s-worker-01 (172.16.101.2) became unreachable on 2025-12-02 ~03:49 MSK, causing multiple pods to be stuck in Terminating state.

Symptoms

• Node status: NotReady
• Kubelet stopped posting node status at 03:49
• Node unreachable via ping (100% packet loss)
• 19 pods stuck on the node, most in Terminating state
• transmission-0 stuck in Terminating for 43+ hours
• StatefulSet shows 0/1 ready replicas

Affected Services

• transmission (StatefulSet, stuck in Terminating)
• argocd-server
• argocd-notifications-controller
• grafana-deployment
• papermc (StatefulSet)
• authelia
• cloudflare-tunnel
• hello (test deployment)
• longhorn instance-manager

Investigation Done

1. Node conditions show NodeStatusUnknown since 03:49
2. Network unreachable (ping timeout to 172.16.101.2)
3. All pods on node are in Terminating or problematic state
4. Memory was overcommitted at 116% on this node

Potential Root Causes

Based on cluster history (CLAUDE.md errors):

1. NFS mount freeze - transmission-config PVC uses StorageClass truenas which no longer exists in cluster. Mount options unknown, possibly hard mount causing D-state freeze.

2. Watchdog disabled - Per Error 13 in CLAUDE.md, watchdog was disabled on worker node due to previous reboot loop issues. No automatic recovery on hang.

3. OOM / Kernel panic - Node memory was 116% overcommitted (9212Mi limits vs 8124Mi available).

4. Hardware failure - Raspberry Pi specific (SD card, power, thermal).

Next Steps

• Physical access to check node status (power, LEDs)
• Reboot node and check journalctl --boot=-1 for crash logs
• Verify transmission-config PV mount options (hard vs soft)
• Consider re-enabling watchdog with proper configuration
• Review memory limits for pods on worker node

Related

• Error 14 in CLAUDE.md: NFS hard mount causing node freeze
• Error 13 in CLAUDE.md: Watchdog configuration issues

[lexfrei]

Root Cause Analysis

Investigation of journalctl --boot=-1 revealed the failure chain:

Timeline

1. 00:49:40 - NFS server (172.16.10.19 / TrueNAS) became unavailable

   nfs: server 172.16.10.19 not responding, timed out
   

2. 00:49:40 - 03:57 - Continuous NFS timeouts (hundreds of messages)

3. ~03:49 - Kubelet stopped posting node status to API server

   • All API calls started timing out (TLS handshake timeout, context deadline exceeded)
   • Node marked as NotReady by control plane

4. ~14:44 - Last log entries before manual reboot

Key Findings

1. NFS Failure was the trigger - TrueNAS NFS server (172.16.10.19) became unresponsive at 00:49:40

2. transmission-config PV has NO soft mount options:

   mountOptions:
   - nfsvers=4.1
   # NO soft, NO timeo - uses hard mount by default!

   This PV was created with old StorageClass truenas (now deleted), which didn't have soft mount options.

3. transmission-downloads-nfs PV has correct soft mount (per manifests/transmission/downloads-pv.yaml):

   mountOptions:
     - soft
     - timeo=100

4. RCU stalls detected throughout the previous boot session:

   rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { P1346947 }
   

   Process P1346947 was causing repeated RCU stalls from Nov 30 through Dec 2.

5. No D-state blocks logged - unlike Error 14 (Loki NFS freeze), this time the kernel didn't report blocked tasks, suggesting the freeze was more gradual.

6. Longhorn iSCSI also had issues:

   EXT4-fs (sdb): shut down requested (2)
   device offline error, dev sdb, sector 1048576 op 0x1:(WRITE)
   Buffer I/O error on dev sdb
   

Root Cause

NFS hard mount on transmission-config PV combined with TrueNAS unavailability caused kubelet to become unresponsive. The kubelet couldn't update node status or respond to API server, leading to node being marked NotReady.

Unlike Error 14 (Loki), this didn't cause a hard kernel freeze - the node remained technically alive but kubelet was stuck waiting for NFS operations that never completed.

Required Fixes

1. Recreate transmission-config PV with soft mount options:

   mountOptions:
     - nfsvers=4.1
     - soft
     - timeo=100
     - retrans=2

2. Investigate why TrueNAS became unavailable at 00:49

3. Consider re-enabling watchdog on worker - it could have rebooted the node and recovered automatically

4. Add NFS server monitoring/alerting

[lexfrei]

CORRECTION: Root cause is NOT NFS

After deeper analysis, the timeline shows:

Actual Timeline

1. 00:49:03 - API Server became unreachable FIRST

   Server 172.16.101.1:6443@ACTIVE*->FAILED from failed dial
   

2. 00:49:04 - All kubelet API calls started timing out

   net/http: TLS handshake timeout
   context deadline exceeded
   

3. 00:49:40 - NFS timeouts started (37 seconds AFTER API server failure)

   nfs: server 172.16.10.19 not responding, timed out
   

Updated Analysis

NFS was NOT the cause - it was a symptom of a broader network failure.

Something caused network connectivity issues at 00:49:03 that affected:

• API Server (172.16.101.1:6443)
• NFS Server (172.16.10.19)

Possible causes:

1. Network switch/router failure
2. Worker node network interface issue
3. Cilium networking problem
4. Physical network cable issue

Key Finding

All NFS mounts already have soft mount option - this was confirmed:

mountOptions:
- nfsvers=4.1
- soft          # ← Present
- timeo=100

So soft mount was working correctly (logging timeouts instead of hanging).

Next Steps

• Check network switch logs around 00:49
• Check if TrueNAS (172.16.10.19) was actually down or just unreachable from worker
• Review Cilium agent logs on worker node around 00:49
• Check physical network connectivity (cables, switch ports)

[lexfrei]

Why Node Did Not Self-Recover

Key Finding

Network physically did not go down - no Link is Down kernel messages on Dec 2.

At 00:48:53, all HTTP/2 watch connections simultaneously received http2: client connection lost. This caused:

1. K3s agent marked API server as FAILED
2. Agent tried to reconnect but kept failing
3. Health checks passed but full TCP connections failed

Evidence

00:49:04 Server 172.16.101.1:6443@FAILED*->RECOVERING from successful health check
00:49:04 Server 172.16.101.1:6443@RECOVERING*->FAILED from failed dial

Health check (TCP SYN-ACK) works, but full connection fails. This points to:

• Stale connection state (conntrack/BPF)
• K3s agent tunnel corruption
• TCP state machine issue

Why No Auto-Recovery

1. Watchdog disabled on worker node (Error 13 in CLAUDE.md)
2. K3s agent doesn't restart on prolonged connection failures
3. Cilium BPF state persists - might hold stale connections
4. No kubelet liveness mechanism to self-heal

Recommendations

1. Enable watchdog on worker with proper configuration
2. Add systemd watchdog for k3s-agent (WatchdogSec= in unit file)
3. Investigate conntrack during next incident: conntrack -L | grep 6443
4. Consider k3s agent restart policy - systemd can restart on network failures
5. Add periodic connectivity check that reboots if API unreachable for >10min

[lexfrei]

Final Investigation Summary

What We Know

1. Timeline: At 00:48:53, all HTTP/2 connections from worker → API server dropped simultaneously (http2: client connection lost)

2. Physical link did NOT go down - no Link is Down kernel messages

3. Control Plane was fine - only saw one read tcp...i/o timeout from worker direction

4. Symptom: Health checks passed, but full TCP connections failed to establish (RECOVERING→FAILED loop)

5. Why no auto-recovery:

   • K3s agent doesn't restart itself on prolonged network failures
   • Watchdog disabled on worker (Error 13)
   • No self-healing mechanism for kubelet

What We Don't Know

• Root cause of connection drop - unclear why all connections died simultaneously
• Possible causes:
  • Kernel 6.17 bug (macb network driver for Raspberry Pi 5 - rp1 support is new)
  • Cilium BPF datapath corruption
  • Transient memory pressure affecting network buffers

Current Kernel

6.17.0-1004-raspi (latest available, no updates)

Both nodes on same switch, so physical network issues would affect both - but only worker failed.

Next Steps to Reproduce/Prevent

1. Add monitoring for:

   • nf_conntrack_count approaching max
   • Network interface errors (ip -s link)
   • Cilium agent health
   • API server connectivity from each node

2. Collect diagnostics on next incident:

   conntrack -L | head -100
   ip neigh show
   cilium status
   dmesg | tail -100
   ss -tunap | grep 6443

3. Consider kernel downgrade to 6.14 to test if issue is 6.17-specific

4. Enable watchdog on worker for automatic recovery on prolonged failures

5. Add k3s-agent restart mechanism when API server unreachable for >10 minutes

[lexfrei]

December 4, 2025 - Control Plane Node Incident

Same pattern now observed on Control Plane node (k8s-cp-01 / 172.16.101.1)

Timeline

Time	Event
10:20:27	RCU stall detected - system already unstable
10:31:27	Worker starts losing connection to CP (lease update timeout)
10:31:29	http2: client connection lost - HTTP/2 connections dropped
10:31:39	Server 172.16.101.1:6443 marked as FAILED
10:31:40 - 10:32:05	FAILED↔RECOVERING flapping (unstable connection)
10:32:10	read tcp...i/o timeout from worker
10:32:18	dial tcp 172.16.101.1:6443: connect: no route to host - complete loss
10:32:48	NFS timeouts begin (symptom, not cause)
10:37:00	RCU stalls resume after partial recovery
~11:30	Hard power cycle applied

Key Differences from Previous Incident

1. This time CP node was affected, not worker
2. No link down events in kernel logs - physical layer stayed up
3. 16-second gap in logs between 10:32:10 and 10:32:26 on CP node (suspicious)
4. Power cycle only - no cable changes - fixed the issue

RCU Stall Statistics

Previous boot (Dec 2 - Dec 4): 99 RCU stalls total
Current boot (26 minutes): 57 RCU stalls already

This is more than 2 RCU stalls per minute - clearly abnormal.

Hardware Context

• Raspberry Pi 5 with RP1 southbridge (new PCIe-based I/O chip)
• Kernel 6.17.0-1004-raspi (Ubuntu 25.10) - bleeding edge with native RP1 support
• Ethernet via Cadence GEM (macb driver) connected through RP1
• RP1 Firmware: eb39cfd516f8c90628aa9d91f52370aade5d0a55

Hypothesis

RP1/macb driver bug in kernel 6.17 causes network hang under load:

1. High load from k3s/etcd operations
2. RP1 ethernet controller enters bad state
3. PHY link stays UP (no carrier loss logged)
4. DMA or IRQ handling stops working
5. No packets transmitted/received despite "active" link
6. RCU stalls may be related to blocked network operations
7. Only power cycle resets RP1 state

Evidence Supporting Driver Bug

• No eth0: Link is Down events
• No errors in ethtool -S eth0
• No thermal/voltage issues
• Switch uptime shows no reboot (1d 15h)
• Power cycle (not cable) fixes issue
• Same pattern on both nodes (both Raspberry Pi 5)
• Frequent RCU stalls suggest kernel instability

Recommendations

1. Consider kernel downgrade to stable raspi kernel (6.11.x or 6.14.x)
2. Monitor for RP1/macb related bug reports in:
   • https://github.com/raspberrypi/linux/issues
   • https://bugs.launchpad.net/ubuntu/+source/linux-raspi
3. Enable hardware watchdog - would have auto-recovered
4. Add periodic network connectivity check + forced reboot on prolonged failure

[lexfrei]

Mitigation Applied: CPU Governor set to Performance

Switched CPU governor from ondemand to performance on both nodes via Ansible.

Before (ondemand)

• 57 RCU stalls in 26 minutes (~2.2 per minute)
• CPU constantly switching between 1.5GHz and 2.4GHz under load

After (performance)

• 0 RCU stalls in 5 minutes after switch
• CPU locked at 2.4GHz

Implementation

Commit: 8e9ece5

• Added cpu-governor.yml task to node-prep role
• Settings persisted via /etc/default/cpufrequtils
• Default governor: performance

Usage

# Apply only CPU governor change
ansible-playbook playbooks/setup-nodes.yaml --tags cpu-governor

# Or run full node-prep
ansible-playbook playbooks/setup-nodes.yaml

Theory

Frequency scaling on Raspberry Pi 5 with RP1 southbridge may interact poorly with kernel 6.17's scheduler, causing:

1. CPU frequency transitions during network IRQ handling
2. Latency spikes in interrupt processing
3. RCU callback delays leading to stalls
4. Potential cascading effect on network driver (macb/Cadence GEM)

Will monitor for stability. If RCU stalls return or network hangs recur, kernel downgrade remains the next option.

[lexfrei]

Investigation Update: RCU Stalls and CPU Frequency Scaling (Dec 4, 2025)

Root Cause Hypothesis

The network hangs appear to be related to CPU frequency transitions interacting poorly with the RP1/macb ethernet driver. Key evidence:

RCU Stall Statistics:

• Previous boot (Dec 2-4): 99 RCU stalls total
• With ondemand governor: 57 RCU stalls in 26 minutes (~2.2/min)
• After switching to performance governor: 0 RCU stalls in 5+ minutes

Important Timing Context

Problems started appearing around upgrade to Ubuntu 25.10 (kernel 6.17). This coincided with Cilium CNI migration (heavy eBPF usage). We cannot definitively attribute to kernel alone - no similar Cilium reports on other hardware found.

Systems fully updated (phased updates ignored), situation unchanged for ~1 month.

Reproduction Notes

• Hangs occur during LOW activity periods, not under load
• Network is NOT heavily loaded when hangs occur
• Load testing could NOT trigger the hang
• Cannot reproduce on demand

Hardware Details

Storage (rules out slow I/O):

• CP: Samsung SSD 990 EVO 1TB via Geekworm X1002 NVMe HAT
• Worker: WD Green SN350 500GB via Geekworm X1002 NVMe HAT

Workaround Being Tested

echo "performance" | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

This eliminates RCU stalls. Long-term stability testing ongoing - not yet confirmed if it prevents hangs.

Related Issues

• Onboard Ethernet on Raspberry Pi 5 experiences packet loss in LAN environment raspberrypi/linux#6855 (EEE packet loss) - different: self-recovers, no RCU stalls
• RX packet drops on Raspberry Pi 5 not reflected in ethtool or softnet_stat raspberrypi/linux#6898 (RX drops) - different: packet loss only, not complete hang