How to use DeepSeek LLM via Azure?

[aiwalter]

How to use DeepSeek LLM in via Azure? Is this already implemented?

[dosubot[bot]]

Yes, the integration of DeepSeek LLM with Azure is implemented. The codebase includes Azure-specific dependencies and configurations for using DeepSeek LLM with Azure services. You can set up DeepSeek LLM on Azure by configuring the DEEPSEEK_API_KEY environment variable, along with other Azure-specific environment variables like AZURE_INFERENCE_ENDPOINT and AZURE_INFERENCE_CREDENTIAL for Azure AI model inference ^[1][2].

Additionally, the codebase uses DefaultAzureCredential from the azure.identity module for Azure-specific authentication and credentials management ^[3][4].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[aiwalter]

@dosubot how would I use deepseek with JSON mode via Azure?

[dosubot[bot]]

To use DeepSeek LLM in JSON mode via Azure, you can follow these steps:

1. Prompt Configuration: Set up a prompt that instructs the LLM to generate a valid JSON following a given schema. This involves creating a ChatPromptTemplate with message templates that guide the LLM to produce JSON output.

2. Response Format: Ensure that the response format is set to {"type": "json_object"} to guarantee that the output is in JSON format.

3. Azure Integration: Utilize Azure-specific configurations and dependencies. This includes setting environment variables like DEEPSEEK_API_KEY, AZURE_INFERENCE_ENDPOINT, and AZURE_INFERENCE_CREDENTIAL for Azure AI model inference.

4. Error Handling and Logging: Implement error handling and logging mechanisms to manage JSON parsing errors and other potential issues during the integration with Azure.

The codebase includes utility functions and classes for handling JSON mode with Azure, such as serialize and deserialize functions for JSON serialization and deserialization, and the AzureKVStore class for managing data storage in Azure ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[aiwalter]

@dosubot how would I use deepseek via Azure as a Chat Model ?

[dosubot[bot]]

To use DeepSeek LLM via Azure as a Chat Model, follow these steps:

1. Install the necessary package:

   pip install llama-index-llms-deepseek

2. Set up the DeepSeek class in your Python code:

   from llama_index.llms.deepseek import DeepSeek
   
   # Initialize the DeepSeek class with the required model and API key
   llm = DeepSeek(model="deepseek-chat", api_key="your_api_key")
   
   # Use the complete method to get a response
   response = llm.complete("Explain the importance of low latency LLMs")
   
   print(response)

3. Configure Azure environment variables:

   • DEEPSEEK_API_KEY: Your DeepSeek API key.
   • OPENAI_API_VERSION: Set this to 2023-05-15.
   • AZURE_OPENAI_ENDPOINT: Your Azure endpoint, e.g., https://YOUR_RESOURCE_NAME.openai.azure.com/.
   • AZURE_OPENAI_API_KEY: Your Azure API key if the API type is azure.

Make sure to replace "your_api_key" with your actual API key for DeepSeek. This setup will enable you to use DeepSeek as a chat model via Azure ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[scdnapier]

@aiwalter were you successful using the Azure DeepSeek model deployment?

[ravwojdyla]

I believe you can use AzureAICompletionsModel

from llama_index.llms.azure_inference import AzureAICompletionsModel

llm = AzureAICompletionsModel(
  model_name="DeepSeek-R1",
  endpoint="https://<FOUNDRY ENDPOINT>.services.ai.azure.com/models", 
  credential=os.environ["AZURE_INFERENCE_CREDENTIAL"],
)

Here's some doc:

• https://docs.llamaindex.ai/en/stable/examples/llm/azure_inference/
• https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-deepseek

[scdnapier]

worked, thanks!

[aiwalter]

@dosubot how to use this with JSON output mode?

[dosubot[bot]]

To use the AzureAICompletionsModel with JSON output mode, you can update the model_kwargs to specify the response format as a JSON object. Here's an example:

from llama_index.llms.azure_inference import AzureAICompletionsModel
from llama_index.llms.azure_inference import ChatMessage, MessageRole
import json

llm = AzureAICompletionsModel(
    model_name="DeepSeek-R1",
    endpoint="https://<FOUNDRY ENDPOINT>.services.ai.azure.com/models",
    credential=os.environ["AZURE_INFERENCE_CREDENTIAL"],
)

llm.model_kwargs.update({"response_format": {"type": "json_object"}})
response = llm.chat(
    [
        ChatMessage(
            role="system",
            content="You are a helpful assistant. When you are asked about if this "
            "is a test, you always reply 'Yes, this is a test.' in a JSON object with "
            "key 'message'.",
        ),
        ChatMessage(role="user", content="Is this a test?"),
    ],
    temperature=0.0,
    top_p=1.0,
)

assert response.message.role == MessageRole.ASSISTANT
assert (
    json.loads(response.message.content.strip()).get("message")
    == "Yes, this is a test."
)

This code snippet demonstrates how to configure the AzureAICompletionsModel to return responses in JSON format by updating the model_kwargs with the desired response format ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[aiwalter]

@dosubot I have this:

llm = AzureAICompletionsModel(
  model_name="DeepSeek-R1",
  endpoint=AZURE_DEEPSEEK_ENDPOINT,
  credential=AZURE_DEEPSEEK_API_KEY,
)
llm.model_kwargs.update({"response_format": {"type": "json_object"}})
out = llm.complete("Hi? output as JSON")

Getting error: ValueError: Unsupported response_format {'type': 'json_object'}
Version llama-index==0.11.10

[dosubot[bot]]

To resolve the "ValueError: Unsupported response_format {'type': 'json_object'}" error with AzureAICompletionsModel in llama-index==0.11.10, ensure that the model supports the json_object format. The chat method supports this format, but the complete method might not. You can try using the chat method instead, as shown in the example below:

from llama_index.llms.azure_inference import AzureAICompletionsModel
from llama_index.llms.azure_inference import ChatMessage, MessageRole
import json

llm = AzureAICompletionsModel(
    model_name="DeepSeek-R1",
    endpoint=AZURE_DEEPSEEK_ENDPOINT,
    credential=AZURE_DEEPSEEK_API_KEY,
)

llm.model_kwargs.update({"response_format": {"type": "json_object"}})
response = llm.chat(
    [
        ChatMessage(
            role="system",
            content="You are a helpful assistant. When you are asked about if this "
            "is a test, you always reply 'Yes, this is a test.' in a JSON object with "
            "key 'message'.",
        ),
        ChatMessage(role="user", content="Hi? output as JSON"),
    ],
    temperature=0.0,
    top_p=1.0,
)

assert response.message.role == MessageRole.ASSISTANT
print(json.loads(response.message.content.strip()).get("message"))

This approach uses the chat method, which is known to support the json_object response format ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[aiwalter]

@dosubot how can I deactivate the thinking output?

I get as output: '<think>\n\n</think>\n\nHello! How can I assist you today? 😊'

But in this case I would just want to have 'Hello! How can I assist you today? 😊' , what parameter to set?

[dosubot[bot]]

I couldn't find specific information about a parameter to deactivate the <think> tags in the output of the AzureAICompletionsModel. However, you can try post-processing the output to remove these tags manually. Here's a simple example of how you might do this:

output = "<think>\n\n</think>\n\nHello! How can I assist you today? 😊"
clean_output = output.replace("<think>\n\n</think>\n\n", "")
print(clean_output)  # Output: Hello! How can I assist you today? 😊

This approach removes the <think> tags from the output string. If the tags are consistently formatted, this method should work effectively.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}