Fix websocket poison connection and leaks on failed requests

[basfroman]

Problem

When many requests share one AsyncSubstrateInterface (one websocket), sometimes it start to fail with Max retries exceeded. and never recover by itself. There are 3 bugs in the websocket layer:

1. Poison pill (dead handler on an open socket).
   The background task _send_recv_task (the _handler) can finish and return TimeoutError("Max retries exceeded.") after it used all retries. But the socket stays in OPEN state. The code restart the handler only when the state is not OPEN or CONNECTING, so nobody restart it. After that every retrieve just re-raise the saved error from the dead task. So the connection is stuck forever and every next call fail instantly with the same error.

2. _waiting_for_response counter leak.
   This counter is +1 when we start to wait for a response, but -1 is done only on success. If retrieve raise, we exit the async with block and skip the -1. So the counter stays wrong (bigger than 0). This breaks the normal state of the connection, for example the auto shutdown logic that use this counter.

3. max_subscriptions semaphore permit leak.
   send take one permit. On success retrieve give it back. But if the request fail, the permit is never returned. After enough failed requests there is no free permit and new requests block.

Fix

On enter, if _send_recv_task is done (dead) but the socket is still OPEN, we force a clean reconnect under the lock (fresh socket and fresh handler). This recover the poison pill on the next call instead of failing forever.

We wrap the request loop in try/finally, so mark_response_received is always called (counter stays balanced), and for every request that did not get a response we call the new discard_request. discard_request give back the semaphore permit, drop the pending future, and keep the id reserved so a late answer from the node can not go into a wrong request.

Tests

e2e test: a small async proxy that can go silent (stop forwarding frames but keep the socket open). It reproduce the real poison pill against a local node, and check that after resume the next call recover. Also checked red and green.

How this is connected to the tao.app relay problem

The tao.app relay service use one shared long living AsyncSubstrateInterface (a singleton, with ws_shutdown_timer=None) for all stake operations that go through MEV plus relay. When this shared websocket hit the poison pill, the background handler was dead but the socket stayed OPEN, so the relay never reconnect and every next stake submit failed instantly. The relay return this to the wallet as 500 Internal server error: Max retries exceeded.. Because it is one shared connection for everybody, one poison event break staking for all users until the service is restarted.

The two leaks are the reason it can not get out of this state by itself. After a failed request _waiting_for_response stays bigger than 0, and the auto shutdown logic only close an idle connection when this counter is <= 0. So the connection recycling that ws_shutdown_timer would normally do is blocked exactly in the poisoned state. The semaphore permit leak is the other side: after enough failed requests there is no free permit and new requests can not be sent at all. For this reason just changing ws_shutdown_timer=None to some value in the relay would not fix the problem (the leaked counter blocks the auto close), it would only add reconnect and re-init cost. The real fix has to be in the library, and this PR make the shared connection heal itself on the next call and stop both leaks, so the relay stake operations stop failing with Max retries exceeded..

[ibraheem-abe]

AB1🤖😂

[thewhaleking]

Probably fine. Cannot look in depth now