Xinference是否支持部署 modelscope社区的OnniSQL模型？

[yangyang9966]

No description provided.

[qinxuye]

可以尝试用自定义模型运行。

如果希望用这个模型的人多，考虑贡献到内置模型。

[yangyang9966]

可以尝试用自定义模型运行。

如果希望用这个模型的人多，考虑贡献到内置模型。

使用自定义模型运行，注册成功后，启动报错，模型文件就是从modescope上的。报错如下：

ValueError: Model not found, name: OmniSQL-7B, format: pytorch, size: 7, quantization: none
2025-03-16 13:41:49,192 xinference.core.worker 1393209 ERROR [request 5a841152-0229-11f0-b2b0-0050569e3666] Leave launch_builtin_model, error: Model not found, name: OmniSQL-7B, format: pytorch, size: 7, quantization: none, elapsed time: 0 s
Traceback (most recent call last):
File "/root/miniconda3/envs/py311/lib/python3.11/site-packages/xinference/core/utils.py", line 93, in wrapped
ret = await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/py311/lib/python3.11/site-packages/xinference/core/worker.py", line 894, in launch_builtin_model
model, model_description = await asyncio.to_thread(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/py311/lib/python3.11/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/py311/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/py311/lib/python3.11/site-packages/xinference/model/core.py", line 73, in create_model_instance
return create_llm_model_instance(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/py311/lib/python3.11/site-packages/xinference/model/llm/core.py", line 219, in create_llm_model_instance
raise ValueError(
ValueError: Model not found, name: OmniSQL-7B, format: pytorch, size: 7, quantization: none

[yangyang9966]

可以尝试用自定义模型运行。
如果希望用这个模型的人多，考虑贡献到内置模型。

使用自定义模型运行，注册成功后，启动报错，模型文件就是从modescope上的。报错如下：

ValueError: Model not found, name: OmniSQL-7B, format: pytorch, size: 7, quantization: none 2025-03-16 13:41:49,192 xinference.core.worker 1393209 ERROR [request 5a841152-0229-11f0-b2b0-0050569e3666] Leave launch_builtin_model, error: Model not found, name: OmniSQL-7B, format: pytorch, size: 7, quantization: none, elapsed time: 0 s Traceback (most recent call last): File "/root/miniconda3/envs/py311/lib/python3.11/site-packages/xinference/core/utils.py", line 93, in wrapped ret = await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/py311/lib/python3.11/site-packages/xinference/core/worker.py", line 894, in launch_builtin_model model, model_description = await asyncio.to_thread( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/py311/lib/python3.11/asyncio/threads.py", line 25, in to_thread return await loop.run_in_executor(None, func_call) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/py311/lib/python3.11/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/py311/lib/python3.11/site-packages/xinference/model/core.py", line 73, in create_model_instance return create_llm_model_instance( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/root/miniconda3/envs/py311/lib/python3.11/site-packages/xinference/model/llm/core.py", line 219, in create_llm_model_instance raise ValueError( ValueError: Model not found, name: OmniSQL-7B, format: pytorch, size: 7, quantization: none

有遇到过自定义模型，注册成功后，启动提示找不到模型的问题不？

[qinxuye]

怎么注册的能看下吗

[yangyang9966]

怎么注册的能看下吗

#3071 (comment)

怎么注册的能看下吗

#3089 ， 和我这个提的问题是一样的。是通过pip部署的集群模型，注册模型成功了，但是启动提示找不到

[qinxuye]

是集群吗？

[yangyang9966]

是集群吗？

嗯嗯，是的，用pip 部署的2节点集群

[qinxuye]

分布式注册注意模型可能要和机器绑定，运行的时候也要指定worker_ip。

[yangyang9966]

分布式注册注意模型可能要和机器绑定，运行的时候也要指定worker_ip。

启动的时候已指定 worker_ip，自定义注册 自己微调的模型以及OmniSQL模型 是成功，但是启动都报一样的错。执行的命令如下：
1.xinference register --model-type LLM --file model.json --persist --endpoint "http://10.0.xxx.xxx:9997/"
2.xinference registrations --endpoint "http://10.0.xxx.xxx:9997/"
3.xinference launch --model-name DeepSeek-R1-1_5B-Distill-TJJ --model-format pytorch --model-engine Transformers --n-worker 1 --worker-ip 10.0.xxx.xxx --endpoint "http://10.0.xxx.xxx:9997/"

model.json