Client disconnect does not cancel upstream proxy request to model runner

[marekyggdrasil]

Summary

When a client aborts a request to a Virtual Model Runner (e.g. cancels the HTTP call, closes the connection, or kills the task), Conductor continues waiting on the upstream model runner until the configured timeout (or until the upstream finishes). The upstream request is not cancelled, so in-flight capacity stays occupied and the AI machine keeps working on a request nobody will read.

Expected behavior

On client disconnect/abort, Conductor should:

1. Cancel the upstream HttpClient call to the selected model runner endpoint.
2. Release in-flight / capacity accounting promptly.
3. Stop waiting for a response that will never be delivered to the client.

Actual behavior

1. Client sends a completion (or similar) request through a VMR.
2. Conductor routes to an endpoint and the AI machine starts work (load is visible).
3. Client cancels/kills the request.
4. Conductor keeps the upstream request open until TimeoutMs (VMR and/or endpoint) or until the upstream completes.
5. Capacity (MaxParallelRequests) and runtime stats remain held for the abandoned request.

Relevant code

Proxy path accepts a CancellationToken and links it with the timeout:

// ProxyController.ForwardRequestAsync
using (CancellationTokenSource timeoutCts =
    new CancellationTokenSource(TimeSpan.FromMilliseconds(effectiveTimeoutMs)))
using (CancellationTokenSource linkedCts =
    CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, timeoutCts.Token))
{
    return await httpClient.SendAsync(
        request,
        HttpCompletionOption.ResponseHeadersRead,
        linkedCts.Token);
}

But the default proxy route invokes the handler without a request-lifetime/abort token:

// ConductorServer (DefaultRoute)
await _ProxyController.HandleRequest(ctx, req).ConfigureAwait(false);
// → cancellationToken defaults to CancellationToken.None

So the linked token only cancels on timeout, never on client disconnect.

Watson Webserver has improved disconnect / request-abort behavior in recent versions; that signal does not appear to be wired into Conductor’s proxy path today.

Suggested fix direction

1. Pass the request abort / lifetime CancellationToken from the Watson HttpContextBase into ProxyController.HandleRequest.
2. Keep linking it with the existing timeout CTS in ForwardRequestAsync (already implemented).
3. Ensure streaming response paths also honor the same token when writing to the client.
4. On cancel, ensure in-flight counters / runtime stats are decremented in finally (largely already present) and that cancel is distinguished from true timeout where useful for metrics/history.

Impact

• Wasted GPU/CPU on abandoned generations.
• Slower recovery of MaxParallelRequests capacity under cancel-heavy clients.
• Misleading load / saturation while clients have already given up.

Workaround

Lower VMR and endpoint TimeoutMs so abandoned requests die sooner. This does not provide true cancel-on-disconnect.