Skip to content

CreateCompletionStreaming stream.Current fails on the last DONE chunk. #24

@KodieGlosserIBM

Description

@KodieGlosserIBM

In vllm and inferencing providers there is a single [DONE] that gets sent as the last chunk in a streaming request.

The issue is that stream.Next() is assuming that all chunks are in json format and tries to unmarshal it:

s.err = json.Unmarshal(s.decoder.Event().Data, &nxt)

This causes the next function to error.

Expected results is that the DONE chunk is more gracefully handled.

Workaround is that we need to see if the syntax error is related to json, which is not a great solution since there could be other reasons the chunk fails to unmarshal.

	for stream.Next() {
		chunk := stream.Current()
		....
	}

	if err := stream.Err(); err != nil {
		// Check if it's a JSON syntax error from the [DONE] marker
		var syntaxErr *json.SyntaxError
		if errors.As(err, &syntaxErr) {
			// The [DONE] marker causes: "invalid character 'D' looking for beginning of value"
			// This is expected and indicates successful stream completion
			return nil
		}
		return err
	}

What is the recommend way to handle this?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions