Feature Request: Add DeepSeek-V3.1

[createthis]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

DeepSeek V3.1

Motivation

Use DeepSeek V3.1

Possible Implementation

#15495

NOTE: --chat-template-kwargs '{"thinking": true}' Doesn't work properly, but non-thinking mode seems to work.

[CISC]

No changes necessary for DeepSeek v3.1 in non-thinking mode, it converts and runs just fine.

The thinking bit requires either updating common_chat_params_init_deepseek_r1 or making a new one for 3.1 (probably best with the latter as it could break R1) so that it handles thinking, like the others (look at Granite f.ex.).

[createthis]

Just FYI for anyone interested, this is an example of the --verbose output when --jinja and --chat-template-kwargs '{"thinking": true}' are given, during a brief turn with open hands ai:

"__verbose": { "index": 0, "content": "First, the user says: \"This is a test. Don't do anything, just respond with 'ok' and finish the task.\"\n\nI need to respond with \"ok\" and then finish the task. The user explicitly said not to do anything else, so I shouldn't execute any commands or make any changes.\n\nLooking at the example, the assistant used various functions like execute_bash, str_replace_editor, and finish. But in this case, the instruction is clear: just respond with \"ok\" and finish.\n\nThe finish function is used to signal completion, so I should use that after responding with \"ok\".\n\nHowever, the user said \"respond with 'ok'\", which might mean I need to output \"ok\" as part of the conversation, not necessarily through a function call. But the instructions say that if there's no function call available, I should answer like normal, but in this context, I think I need to use the finish function to end the task.\n\nRe-reading the user's message: \"just respond with 'ok' and finish the task.\" So, I should probably output \"ok\" in the conversation and then call finish.\n\nBut the format requires that if I choose to call a function, I must reply in the specified format with no suffix. Also, I can provide reasoning before the function call.\n\nSince the user said \"don't do anything\", I shouldn't execute any other functions like bash commands or file edits. Just respond with \"ok\" and finish.\n\nI think the best way is to first output \"ok\" as a natural language response, and then call the finish function to end the task.\n\nThe instructions say: \"If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls.\" But in this case, I am going to use the finish function, so I should call it.\n\nPerhaps I can just call finish with a message that includes \"ok\".\n\nLet me check the finish function parameters: it requires a message, which is the final message to send to the user.\n\nSo, I can set the message to \"ok\" to respond with that and finish.\n\nThe user said \"respond with 'ok' and finish the task\", so using finish with message \"ok\" should accomplish both.\n\nI shouldn't do anything else, like executing bash or other functions, as per \"don't do anything\".\n\nSo, my response should be a single function call to finish with message \"ok\".</think>ok\n\n<function=finish>\n<parameter=message>ok</parameter>\n</function>",

Simplified, that is:

        "__verbose": {
            "index": 0,
            "content": "REASONING</think>ok\n\n<function=finish>\n<parameter=message>ok</parameter>\n</function>",

The full startup command was:

./build/bin/llama-server \
    --model /data2/DeepSeek-V3.1-GGUF/UD-Q4_K_XL/DeepSeek-V3.1-UD-Q4_K_XL-00001-of-00008.gguf \
    --alias DeepSeek-V3.1:671b-q4_k_xl \
    --no-webui \
    --numa numactl \
    --threads 32 \
    --ctx-size 163840 \
    --n-gpu-layers 62 \
    -ot "blk\.(3|4|5|6|7|8|9|10|11)\.ffn_.*=CUDA0" \
    -ot exps=CPU \
    -ub 4096 -b 4096 \
    --seed 3407 \
    --temp 0.6 \
    --top-p 0.95 \
    --min-p 0.1 \
    --log-colors \
    --flash-attn \
    --host 0.0.0.0 \
    --jinja \
    --prio 2 \
    --chat-template-kwargs '{"thinking": true}' \
    --verbose \
    --port 11434

[createthis]

I've got something working in my local branch: createthis#6

Now with open hands ai and --verbose and --chat-template-kwargs '{"thinking": true}' I get:

        "choices": [
            {
                "finish_reason": "stop",
                "index": 0,
                "message": {
                    "role": "assistant",
                    "reasoning_content": "First, the user says: \"This is a test. Don't do anything, just respond with 'ok' and finish the task.\"\n\nI need to respond with \"ok\" and then finish the task. The user explicitly said not to do anything else, so I shouldn't execute any commands or make any changes.\n\nLooking at the example, the assistant used various functions like execute_bash, str_replace_editor, and finish. But in this case, the instruction is clear: just respond with \"ok\" and finish.\n\nThe finish function is used to signal completion, so I should use that after responding with \"ok\".\n\nHowever, the user said \"respond with 'ok'\", which might mean I need to output \"ok\" as part of the conversation, not necessarily through a function call. But the instructions say that if there's no function call available, I should answer like normal, but in this context, I think I need to use the finish function to end the task.\n\nRe-reading the user's message: \"just respond with 'ok' and finish the task.\" So, I should probably output \"ok\" in the conversation and then call finish.\n\nBut the format requires that if I choose to call a function, I must reply in the specified format with no suffix. Also, I can provide reasoning before the function call.\n\nSince the user said \"don't do anything\", I shouldn't execute any other functions like bash commands or file edits. Just respond with \"ok\" and finish.\n\nI think the best way is to first output \"ok\" as a natural language response, and then call the finish function to end the task.\n\nThe instructions say: \"If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls.\" But in this case, I am going to use the finish function, so I should call it.\n\nPerhaps I can just call finish with a message that includes \"ok\".\n\nLet me check the finish function parameters: it requires a message, which is the final message to send to the user.\n\nSo, I can set the message to \"ok\" to respond with that and finish.\n\nThe user said \"respond with 'ok' and finish the task\", so using finish with message \"ok\" should accomplish both.\n\nI shouldn't do anything else, like executing bash or other functions, as per \"don't do anything\".\n\nSo, my response should be a single function call to finish with message \"ok\".",
                    "content": "ok\n\n<function=finish>\n<parameter=message>ok</parameter>\n</function>"
                }
            }
        ],

For this verbose content:

"__verbose": { "index": 0, "content": "First, the user says: \"This is a test. Don't do anything, just respond with 'ok' and finish the task.\"\n\nI need to respond with \"ok\" and then finish the task. The user explicitly said not to do anything else, so I shouldn't execute any commands or make any changes.\n\nLooking at the example, the assistant used various functions like execute_bash, str_replace_editor, and finish. But in this case, the instruction is clear: just respond with \"ok\" and finish.\n\nThe finish function is used to signal completion, so I should use that after responding with \"ok\".\n\nHowever, the user said \"respond with 'ok'\", which might mean I need to output \"ok\" as part of the conversation, not necessarily through a function call. But the instructions say that if there's no function call available, I should answer like normal, but in this context, I think I need to use the finish function to end the task.\n\nRe-reading the user's message: \"just respond with 'ok' and finish the task.\" So, I should probably output \"ok\" in the conversation and then call finish.\n\nBut the format requires that if I choose to call a function, I must reply in the specified format with no suffix. Also, I can provide reasoning before the function call.\n\nSince the user said \"don't do anything\", I shouldn't execute any other functions like bash commands or file edits. Just respond with \"ok\" and finish.\n\nI think the best way is to first output \"ok\" as a natural language response, and then call the finish function to end the task.\n\nThe instructions say: \"If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls.\" But in this case, I am going to use the finish function, so I should call it.\n\nPerhaps I can just call finish with a message that includes \"ok\".\n\nLet me check the finish function parameters: it requires a message, which is the final message to send to the user.\n\nSo, I can set the message to \"ok\" to respond with that and finish.\n\nThe user said \"respond with 'ok' and finish the task\", so using finish with message \"ok\" should accomplish both.\n\nI shouldn't do anything else, like executing bash or other functions, as per \"don't do anything\".\n\nSo, my response should be a single function call to finish with message \"ok\".</think>ok\n\n<function=finish>\n<parameter=message>ok</parameter>\n</function>",

Something is still wrong in streaming mode though. Open WebUI uses streaming and it is duplicating the reasoning content in the output:

Still, at least it's populating reasoning_content now. Progress.

Oh, and my unit test is failing miserably.

I have to task switch but if no one else jumps in I can come back to this later this weekend.

[MB7979]

Is chat-template-kwargs server only, unsupported on main? I’m getting failures with that flag, even with some models that do seem to be supported. As for 3.1, would love to see this implemented as it’s an excellent model with thinking enabled. I’ve managed to test it using no-cnv and manually pasting the chat template to get thinking working.

[CISC]

Is chat-template-kwargs server only, unsupported on main? I’m getting failures with that flag, even with some models that do seem to be supported.

Yes, see #13196 (comment)

[createthis]

This PR makes thinking mode the default for DeepSeek V3.1: #15533

Disable with --reasoning-budget 0.

I've tested the server with --jinja. I haven't tested the cli. I also haven't tested tool calling.

[createthis]

Here's the unsloth DeepSeek-V3.1 chat template in human readable form, just in case anyone wants to look through it: https://gist.github.com/createthis/a360c70d54bdba7c694d4bd49a309515#file-unsloth_chat_template-jinja

And here's the official DeepSeek-V3.1 chat template in human readable form: https://gist.github.com/createthis/95b3921e62b246e1a66724407d1dad58#file-chat_template-jinja

They're very similar.

[createthis]

Tool calls with tool_test.py AND open hands are now working in non-reasoning mode. It is not clear to me if this model is capable of coherent tool calls in reasoning mode. The model card states:

Toolcall is supported in non-thinking mode.

https://huggingface.co/deepseek-ai/DeepSeek-V3.1#toolcall

I also think this model is a bit like GPT OSS in that it really needs to use tool calls. I think it may be too strictly fine tuned and rigid to reliably make tool calls using ad-hoc prompted formats.

[createthis]

thinking tool call

Here's what the model is outputting in thinking mode for tool calls (this one fails):

"__verbose": { "index": 0, "content": "First, the user is asking for the current time in Tokyo. I have a tool called \"get_time\" that can get the current local time for a city. The tool requires a parameter \"city\" which should be a string.\n\nI need to call this tool with the city set to \"Tokyo\". The tool will then return the current time.\n\nAfter calling the tool, I should return the final answer concisely, as per the instructions.\n\nThe tool call syntax is:<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>tool_name<｜tool▁sep｜>{\"arg1\": \"some_value\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜> In this case, the tool name is \"get_time\" and the argument is \"city\" with value \"Tokyo\".\n\nSo, the tool call should be:<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_time<｜tool▁sep｜>{\"city\": \"Tokyo\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜> \n\nOnce I get the response from the tool, I'll format the answer concisely. For example, if the tool returns something like \"2024-06-15 12:34:56\", I should say something like \"The current time in Tokyo is 12:34 PM\" or similar, but keep it concise.\n\nI should not assume the response; I need to wait for the tool's output.\n\nFinally, since this is a function call, I need to output it in the correct format.</think><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_time\n```json\n{\"city\": \"Tokyo\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>",

Looking closer at that last bit:

</think><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>function<｜tool▁sep｜>get_time\n```json\n{\"city\": \"Tokyo\"}\n```<｜tool▁call▁end｜><｜tool▁calls▁end｜>

non-thinking tool call

Compare that to the non-thinking output (this one succeeds):

"__verbose": { "index": 0, "content": "I'll get the current time in Tokyo for you.<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_time<｜tool▁sep｜>{\"city\": \"Tokyo\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>",

Looking closer at that last bit:

<｜tool▁calls▁begin｜><｜tool▁call▁begin｜>get_time<｜tool▁sep｜>{\"city\": \"Tokyo\"}<｜tool▁call▁end｜><｜tool▁calls▁end｜>

Note how the function name is before <｜tool▁sep｜>.

[createthis]

I'll just attempt to support both syntaxes. It looks like that's what the original code was trying to do too.

[createthis]

I've added parsing tests for the above tool call inputs.

I've got a highly experimental branch where both of the above tool input tests pass, but it breaks thinking inference somehow: createthis#7

I'm going to task switch. Not sure when I'll be back to this.

[createthis]

Tool calls now work in thinking and non-thinking modes. I made a copy of parse_json_tool_calls in order to not accidentally introduce regressions in other models. I've introduced a regression in streaming where reasoning content initially comes across as regular content. I need to think about how to address that.

I added several unit tests for tool call parsing. I find these extremely helpful when dealing with edge cases. The danger branch has been merged into the regular branch: #15533

[createthis]

Branch is currently broken. I know what I need to fix and how to fix it. I just have to task switch for a moment.