local model evaluation

[llexieguo]

Hi, thanks for the great project! I’m wondering how to test a locally deployed model like qwen3. The current code seems to enforce API-based calls — is there a way to adapt this for local models?

[black-yt]

Hello, thank you very much for your interest in our project.

You can evaluate open-source models such as Qwen3 locally using vLLM and access them through an OpenAI-compatible API, which is convenient for standardized evaluation.

Concretely, you can install vLLM via pip (pip install vllm) and start the service with, for example:

vllm serve /path/to/Qwen3-8B --tensor-parallel-size 8 --trust-remote-code

Once the service is running, the model can be queried using the OpenAI API format. The base URL typically points to the local service (e.g., http://localhost:8000/v1), and the API key can be set to "EMPTY", for example:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

In addition to vLLM, similar deployment options include SGLang and LMDeploy, which can also be used to expose models via OpenAI-style APIs for evaluation.

If you have any further questions, please feel free to ask.