Strange association between unrelated mining channels

[vnprc]

I have been chasing down a bug for a month or two that causes all reported hashrate to crash to zero and stay there. A restart fixes it. It's been happening ever since someone pointed their nerdminer to the pool. The pattern seems to be that the nerdminer's low hashrate triggers a vardiff bug where the pool tries to negotiate the difficulty level too low and triggers a cascading failure, leading to a channel disconnect loop. This somehow impacts all channels and drops hashrate for the entire pool to zero.

I have tried unsuccessfully to fix this bug many times. But, honestly, I haven't dug in and given it my full attention so I'm obviously missing something.

This time there seems to be a different behavior pattern. Two miners seem to be associated in a failure loop now: THE Dr Nickamoto and the nerdminer (with a testnet address username) are both stuck with zero shares. Actually, in the screenshot the nerdminer found a share. This is very rare because the nerdminer hashrate is absolute garbage. But THE Dr Nickamoto is running a bitaxe and should be finding shares.

This seems to indicate that the bug somehow affects multiple mining channels but not necessarily all of them. It's still a pretty severe failure because all impacted miners are unable to submit shares successfully.

[vnprc]

Also notable is that the nerdminer and all it's entangled mining channels reset at the same time. You can see this in the screenshot because they have sequential channel ID values and the same connected time. They have both been churning through connections together and this is how they got channel IDs up in the 90s.

[vnprc]

aaaaaand now the whole pool is down again. I am getting the impression that there are a number of bugs interacting in strange ways to create this behavior. This is the most visible behavior and the bug I have been chasing for months. It seems to happen after every restart, sometimes within a day, sometimes after a week, but it always happens eventually. Fixed with a simple restart.

[vnprc]

The nerdminer and dr nickamoto are entangled once again. Very curious.

[vnprc]

This time it broke fast, last time it went for over a week. ¯\_(ツ)_/¯

[vnprc]

initial channel id allocation after restart, seems to indicate that dr nickamoto experienced some churn in establishing a channel, we'll see if the quantum entanglement between nickamoto and nerdminer is reestablished

[vnprc]

I don't think they are entangled, but they both seem affected by the same bug that causes them to periodically reset the channel. This is a new development. Dr Nickamoto was mining successfully before the last update. The nerdminer has always struggled to stay connected.