In vllm and inferencing providers there is a single [DONE] that gets sent as the last chunk in a streaming request.
The issue is that stream.Next() is assuming that all chunks are in json format and tries to unmarshal it:
|
s.err = json.Unmarshal(s.decoder.Event().Data, &nxt) |
This causes the next function to error.
Expected results is that the DONE chunk is more gracefully handled.
Workaround is that we need to see if the syntax error is related to json, which is not a great solution since there could be other reasons the chunk fails to unmarshal.
for stream.Next() {
chunk := stream.Current()
....
}
if err := stream.Err(); err != nil {
// Check if it's a JSON syntax error from the [DONE] marker
var syntaxErr *json.SyntaxError
if errors.As(err, &syntaxErr) {
// The [DONE] marker causes: "invalid character 'D' looking for beginning of value"
// This is expected and indicates successful stream completion
return nil
}
return err
}
What is the recommend way to handle this?
In vllm and inferencing providers there is a single
[DONE]that gets sent as the last chunk in a streaming request.The issue is that stream.Next() is assuming that all chunks are in json format and tries to unmarshal it:
llama-stack-client-go/packages/ssestream/ssestream.go
Line 159 in 09986d1
This causes the next function to error.
Expected results is that the DONE chunk is more gracefully handled.
Workaround is that we need to see if the syntax error is related to json, which is not a great solution since there could be other reasons the chunk fails to unmarshal.
What is the recommend way to handle this?