Chat stream shows "Reconnecting" then "Stream interrupted" while the backend response completes successfully

[paullizer]

Issue

Users report that a chat prompt appears to fail — the UI shows "thinking", then "Reconnecting", then a "Stream interrupted:" banner — while the backend actually completed the response and persisted it to Cosmos DB. The finished answer only becomes visible after the user closes the browser and returns, or navigates away and reopens the conversation.

Two users reported this independently.

I've been using simplechat, often with saved prompts. Sometimes not. What frequently happens is, the prompt is sent, and simplechat sits on it, says "thinking" then "reconnecting" then "stream interrupted". I've tried "Retry", but what seems to work is exiting out of the browser (Chrome), then after a few minutes coming back and voila` the results are there.

I get this as well [...] at other times, it does seem to just get "lost" - clicking the logo in the top-left then clicking on the "start chatting" button seems to often resolve it without needing to close out.

The fact that both a browser restart and an in-app navigation recover the answer is the key signal: the backend worker finished successfully. Only the browser's live SSE connection was lost, and the reconnect path failed to rejoin it.

Steps to Reproduce

Not yet reduced to a deterministic repro. Reported conditions:

1. Send a chat prompt (reported both with and without saved prompts).
2. Wait through a long-running response.
3. The placeholder shows "thinking", then flips to "Reconnecting", then renders the "Stream interrupted:" banner.
4. Click "Retry" — the problem is not resolved.
5. Close the browser entirely, wait a few minutes, reopen the conversation. The completed response is present.

Expected Behavior

• A dropped SSE connection should reconnect reliably while the backend run is still active.
• If reconnect cannot be re-established immediately, the client should keep trying with backoff rather than giving up after a single attempt.
• Once the backend finishes and persists the reply, the open tab should show it without requiring a manual page reload or browser restart.
• "Retry" on an interrupted message should rejoin or recover the in-flight run, not silently start a second, duplicate generation.

Actual Behavior

The reconnect attempt fails once and the message is left permanently in an error state, even though the answer exists server-side moments later.

Findings from code review

These are confirmed by reading the code. They are candidate causes; see "Confirmation needed" below for what still has to be measured.

1. A failed reattach is terminal — recovery is attempted exactly once

attemptStreamingRecovery re-enters consumeStreamingResponse with allowRecovery: false:

// application/single_app/static/js/chat/chat-streaming.js:610-627
return consumeStreamingResponse(
    signal => fetch(`/api/chat/stream/reattach/${conversationId}`, { ... }),
    reconnectMessageId,
    tempUserMessageId,
    {
        onDone, onError, onFinally,
        allowRecovery: false,
        ...
    },
);

Because the reattach stream itself runs with allowRecovery: false, any failure on that second connection falls straight through to handleStreamError, which renders the Stream interrupted: banner (chat-streaming.js:322). There is no retry, no backoff, and no second attempt. One unlucky reconnect is enough to permanently fail the message.

2. Reattach depends on worker affinity unless Redis is enabled

ActiveConversationStreamRegistry holds sessions in process-local memory and falls back to the shared cache:

# application/single_app/route_backend_chats.py:8542-8603
class ActiveConversationStreamRegistry:
    def __init__(self, completed_session_ttl_seconds=600, heartbeat_interval_seconds=15):
        self._sessions = {}
        ...
    def get_session(self, user_id, conversation_id, active_only=False):
        session = self._sessions.get(key)
        if not session:
            metadata = app_settings_cache.get_stream_session_meta(f'{user_id}:{conversation_id}')
            if not metadata:
                return None

get_stream_session_meta is Redis-backed only when Redis is enabled. Otherwise app_settings_cache.py binds the *_mem variants (get_stream_session_meta_mem at app_settings_cache.py:1042, get_stream_session_events_mem at 1068), which are per-process dictionaries.

gunicorn.conf.py:25 defaults to workers = 2. So on a Redis-less deployment, the follow-up GET /api/chat/stream/status/<id> and GET /api/chat/stream/reattach/<id> requests are load-balanced and may land on the worker that never held the session. chat_stream_reattach_api then returns 404:

# application/single_app/route_backend_chats.py:24621-24623
stream_session = CHAT_STREAM_REGISTRY.get_session(user_id, conversation_id, active_only=True)
if not stream_session:
    return jsonify({'error': 'No active stream is available for this conversation'}), 404

Combined with finding 1, a single wrong-worker reattach produces a permanent "Stream interrupted".

This is consistent with CHAT_STREAM_HEARTBEAT_REATTACH_FIX.md, which already notes that cross-worker reattach requires Redis and that the non-Redis path is same-process fallback only. What is missing is graceful behavior when that fallback cannot serve the request.

3. Nothing recovers the message after the banner is shown

Once handleStreamError runs, the message is inert. There is no polling of /api/chat/stream/status/<id>, no delayed re-check, and no refetch of persisted messages. The backend finishes and writes the reply to Cosmos, but the already-open tab never learns about it.

This is exactly why closing the browser or clicking the logo and reopening the conversation works — both paths re-fetch persisted messages from scratch.

4. "Retry" starts a new generation instead of reattaching

executeMessageRetry (application/single_app/static/js/chat/chat-retry.js:226) issues a fresh generation request through the retry API and then calls the chat API again. It has no awareness of an in-flight backend run for that conversation.

So clicking "Retry" while the original stream is still executing server-side can start a second concurrent generation rather than rejoining the first. That matches the report that Retry does not help, and it wastes model capacity.

5. Stream session TTL caps the recovery window

The registry uses completed_session_ttl_seconds=600, and ActiveConversationStreamSession defaults to session_ttl_seconds=600 (route_backend_chats.py:8221). A response whose gap between disconnect and reattach exceeds ~10 minutes cannot be recovered through reattach at all, only through a full conversation reload.

Impact

• Users believe a response failed when it actually succeeded, and lose confidence in the product.
• The documented workaround is "close your browser and come back later", which is not acceptable.
• Retry can double-bill model usage by launching a duplicate generation against a run that is still active.
• Most likely to bite long-running responses (agents, tool calls, tabular analysis) and any deployment running multiple gunicorn workers without Redis — which is the default configuration.

Confirmation needed before fixing

The client already reports detailed stream telemetry to POST /api/chat/stream/client-event. Before committing to a fix, pull these event types from App Insights to establish which failure mode actually dominates in production:

• stream_premature_end (with status: done_without_terminal_event)
• stream_read_error
• stream_request_error
• stream_recovery_unavailable (includes pending and reattachable flags)
• stream_recovery_attempt versus stream_recovery_attached

The ratio of stream_recovery_attempt to stream_recovery_attached, and the pending/reattachable values on stream_recovery_unavailable, should show directly whether reattach is failing because of worker affinity, TTL expiry, or something else. Server-side, the [STREAMING] Stream consumer reattached and [STREAMING] Reattached stream consumer detached log events give the matching backend view.

Also worth capturing: whether affected deployments have Redis enabled, and their GUNICORN_WORKERS value.

Suggested Approach

Ordered roughly by value per unit of risk. Steps 1 and 2 are worth doing regardless of what telemetry shows.

1. Make reconnect resilient rather than single-shot. Allow a bounded number of reattach attempts with backoff instead of allowRecovery: false on the first failure. Treat a 404 from /reattach as retryable for a short window rather than immediately fatal.
2. Recover after the banner. When a stream ends in an interrupted state, poll /api/chat/stream/status/<id> (or refetch the conversation) for a bounded period so a reply that lands seconds later replaces the error state automatically.
3. Fix Retry semantics for interrupted messages. If a run is still active for the conversation, Retry should reattach to it rather than starting a duplicate generation.
4. Handle the Redis-less multi-worker case honestly. Either make the stream registry work across workers without Redis, warn administrators when multi-worker plus no Redis makes reattach unreliable, or pin reattach to the owning worker.
5. Revisit the 600s TTL once telemetry shows the real distribution of disconnect-to-reattach gaps.
6. Add functional tests covering: reattach retry/backoff behavior, post-interruption recovery, and Retry against a conversation with an active stream.

Notes

• Reported against version 0.261.003.
• Related but distinct: Post-stream message reload is dead code: window.chatMessages.loadMessages is never exported #1286 (the post-stream window.chatMessages.loadMessages reload guard is dead code). That issue covers plugin-persisted extra messages; this issue covers the reconnect path itself. They may share a fix in step 2, so they should be looked at together.
• The second reporter also correlates the symptom with long prompts. That specific aspect - a generic, unexplained error for oversized input - is filed separately as Oversized prompts fail with a generic "Stream interrupted" instead of a context-length error #1380, since it is an error-classification defect with a different fix.