[BUG]: Can't change the output token limit with novita ai

[nidupb]

How are you running AnythingLLM?

Docker (local)

What happened?

When experimenting with deepseek, Novita seems to limit the token output to 2048 when it should go up to 8196.

Novita provide an example of API implementation with the base limit :

from openai import OpenAI
  
client = OpenAI(
    base_url="https://api.novita.ai/v3/openai",
    api_key="<YOUR Novita AI API Key>",
)

model = "deepseek/deepseek-r1"
stream = True # or False
max_tokens = 2048
system_content = """Be a helpful assistant"""
temperature = 1
top_p = 1
min_p = 0
top_k = 50
presence_penalty = 0
frequency_penalty = 0
repetition_penalty = 1
response_format = { "type": "text" }

chat_completion_res = client.chat.completions.create(
    model=model,
    messages=[
        {
            "role": "system",
            "content": system_content,
        },
        {
            "role": "user",
            "content": "Hi there!",
        }
    ],
    stream=stream,
    max_tokens=max_tokens,
    temperature=temperature,
    top_p=top_p,
    presence_penalty=presence_penalty,
    frequency_penalty=frequency_penalty,
    response_format=response_format,
    extra_body={
      "top_k": top_k,
      "repetition_penalty": repetition_penalty,
      "min_p": min_p
    }
  )

if stream:
    for chunk in chat_completion_res:
        print(chunk.choices[0].delta.content or "", end="")
else:
    print(chat_completion_res.choices[0].message.content)
  

Are there known steps to reproduce?

Ask any complex issue that'll likely prompt an output above 2048 tokens

[Karasowl]

I wanted to report that I reproduced the issue using the standard AnythingLLM setup (not running in Docker) with the deepseek model via Novita AI. The response is still truncated.