Oversized prompts fail with a generic "Stream interrupted" instead of a context-length error

[paullizer]

Issue

When a chat request fails because the input is too large for the model, the user sees a generic "Stream interrupted:" banner with no indication that the prompt was the problem. Nothing tells them the input exceeded the context window, and nothing suggests shortening it.

Reported by a user who worked out the cause by trial and error:

I get this as well - but more specifically it appears that there is an input length limit that will return this error without further context. I have found that reducing my much longer prompts sometimes resolves this issue.

Having a user reverse-engineer a context-length limit from an unexplained error is a bad outcome, and it is entirely avoidable — the streaming error path already knows how to give a specific message for one error class (rate limits) and simply does not do it for any other.

Steps to Reproduce

1. Send a chat prompt with a very large input — a long pasted prompt, a long conversation history, or a large amount of selected document context.
2. The stream fails.
3. The UI renders "Stream interrupted: Something went wrong while streaming the response. Please try again."
4. Nothing indicates the input length caused the failure.
5. Shortening the prompt makes the request succeed.

Expected Behavior

When a request fails because of input size, the user should get an actionable message naming the cause and the remedy — for example that the prompt or conversation history is too long and should be shortened or moved to a new conversation.

More generally, provider 400 responses are user-correctable errors and should not be flattened into the same opaque text used for genuine server faults.

Actual Behavior

Every non-rate-limit failure in the streaming path collapses into one generic message.

Findings from code review

1. The streaming error path classifies only rate limits

Both streaming failure handlers special-case rate limiting and nothing else.

Mid-stream failure:

# application/single_app/route_backend_chats.py:24359-24363
stream_rate_limited = is_rate_limit_error(error_msg, e)
stream_failure_message = (
    get_rate_limit_message() if stream_rate_limited
    else CLIENT_SAFE_STREAM_ERROR_MESSAGE
)

Top-level handler:

# application/single_app/route_backend_chats.py:24511-24518
if is_rate_limit_error(str(e), e):
    yield build_stream_error_event(
        get_rate_limit_message(),
        rate_limited=True,
        status_code=429,
    )
else:
    yield build_stream_error_event()

build_stream_error_event() with no argument uses the default:

# application/single_app/route_backend_chats.py:14270
CLIENT_SAFE_STREAM_ERROR_MESSAGE = 'Something went wrong while streaming the response. Please try again.'

So an Azure OpenAI BadRequestError carrying context_length_exceeded is indistinguishable from any other failure by the time it reaches the browser.

Notably, the comment directly above the rate-limit branch states the intent:

reaches here, so tell the user that plainly rather than letting it look like an unexplained stream failure.

That reasoning is correct and should extend to context-length and other user-correctable 400s. It was just never applied beyond throttling.

2. A provider bad-request classifier already exists but is not wired into chat streaming

# application/single_app/route_backend_chats.py:14297-14298
def is_provider_bad_request_error(error_message, exc):
    return '400' in str(error_message or '') and 'BadRequestError' in str(type(exc))

Its only call sites are the two image generation paths (route_backend_chats.py:16159 and 18362), where it produces a helpful "request was invalid, please edit the prompt" message. The chat streaming path never calls it.

The same is true of is_content_safety_error (14293), which is also only used by image generation (16153, 18356). A content-safety block during a chat stream therefore also surfaces as the generic message — worth fixing in the same change.

3. A friendly context-length message exists, but only on the non-streaming path

# application/single_app/route_backend_chats.py:19886-19887
if "context length" in str(e).lower():
    return ("Sorry, the conversation history is too long even after summarization. "
            "Please start a new conversation or try a shorter message.", gpt_model, None, None, None)

This lives in the GPT fallback used by the non-streaming path. Streaming is the primary chat path, so in practice most users never see this message.

4. get_safe_stream_error_message deliberately discards 5xx detail

# application/single_app/route_backend_chats.py:14279-14284
def get_safe_stream_error_message(payload, status_code, fallback_message):
    if status_code >= 500 and not (
        payload.get('service_health_warning') or payload.get('warning_type')
    ):
        return fallback_message
    return payload.get('error') or fallback_message

Suppressing internal detail on 5xx is correct and should stay. The gap is that genuine 4xx user-correctable errors never get a specific message constructed in the first place, so there is nothing for this function to pass through.

5. No size preflight anywhere

There is no maxlength on the chat input and no server-side validation of user message or assembled context size. The first signal that a request is too large is a failed model call, after the user has already waited.

Impact

• Users cannot tell a too-long prompt from a transient outage, so they retry a request that can never succeed.
• The failure is silent about a limit that users must otherwise discover by bisecting their own prompt.
• Affects the primary chat path, since streaming is the default.
• Content-safety blocks during streaming are equally opaque, for the same reason.

Suggested Approach

1. Classify user-correctable provider errors in the streaming path. Reuse the existing is_provider_bad_request_error and is_content_safety_error helpers in both streaming failure handlers (route_backend_chats.py:24359 and 24511), alongside the existing rate-limit branch.
2. Add explicit context-length detection. Detect context_length_exceeded and "maximum context length" and emit a specific, actionable message. Reuse the wording already proven on the non-streaming path (19886) so both paths agree.
3. Keep the 5xx suppression as-is. This change should widen 4xx classification only, never leak internal 5xx detail.
4. Consider a preflight size check. Estimate the assembled context against the deployment's model limits before dispatching, and fail fast with a clear message rather than after a long wait. Size this separately — it depends on reliable per-model limit metadata.
5. Add a functional test asserting that a simulated provider 400 with context_length_exceeded produces a context-length-specific SSE error event rather than CLIENT_SAFE_STREAM_ERROR_MESSAGE.

Notes

• Reported against version 0.261.003.
• Related: Chat stream shows "Reconnecting" then "Stream interrupted" while the backend response completes successfully #1379 covers the reconnect/reattach failure, filed from the same pair of user reports. That one explains why some of these failures are recoverable by reloading; this one explains why the error text is unhelpful when they are not. They are independent fixes.
• Steps 1-3 are worth doing on their own merits regardless of whether context length turns out to be the dominant cause for this specific reporter — the streaming path currently discards actionable information it already has.