Skip to content

[bug] Cant start with 2x video cards #22

Open
Open
[bug] Cant start with 2x video cards#22
Labels
bugSomething isn't working

Description

@nordkiller51
nordkiller51
opened on May 27, 2026

What happened

I try to start with ampere bat and got error
[GPU mismatch] this zip ships an Ampere/Ada wheel (CUDA 12.6) which has no Blackwell (sm_120) kernels.
Detected: NVIDIA GeForce RTX 5070 (sm_120) | NVIDIA GeForce RTX 3090 (sm_86).
Download the -blackwell zip from
https://github.com/devnen/qwen3.6-windows-server/releases
and re-extract on top of this install.

Then a try to start with blackwell bat - download and start ok. Then a chose ctx127 on gpu0 and got error
Free memory on device cuda:0 (10.78/11.94 GiB) on startup is less than desired GPU memory utilization (0.948, 11.32 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.
So, i edit config to gpu1 and got same error

How can i run it on 3090?

Hardware + OS

Driver Version: 596.49
CUDA Version : 13.2
NVIDIA GeForce RTX 5070
NVIDIA GeForce RTX 3090
win 11 25h2
amd ryzen 5 3600 with 32 gb RAM

display attached to 5070

Install path

  • [v ] Portable launcher zip
  • Wheel-only install into my own venv
  • Built from source

Snapshot you tried

start_127k

Reproducer

(The exact command or click sequence that triggered the failure.)

Logs

Paste the boot section of logs\vllm_server.<port>.log (first ~50 lines)
plus the last ~50 lines around the failure. Don't worry about size, we'd
rather have too much than too little.

============================================================
vLLM serve: qwen3.6-27b-autoround
  Model   : C:\Users\nordk\Desktop\qwen3.6-windows-server\models\Qwen3.6-27B-int4-AutoRound
  Ctx     : 127000  |  TP: 1  |  PP: 1
  KV dtype: fp8_e4m3  |  MTP: True (n=3)
  Listen  : http://0.0.0.0:5001
============================================================
C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Scripts\vllm.exe serve C:\Users\nordk\Desktop\qwen3.6-windows-server\models\Qwen3.6-27B-int4-AutoRound --served-model-name=qwen3.6-27b-autoround --quantization=auto-round --max-model-len=127000 --max-num-seqs=1 --max-num-batched-tokens=4128 --block-size=32 --enable-prefix-caching --enable-chunked-prefill --enable-auto-tool-choice --tool-call-parser=qwen3_coder --reasoning-parser=qwen3 --chat-template=C:\Users\nordk\Desktop\qwen3.6-windows-server\templates\qwen3.5-enhanced.jinja --default-chat-template-kwargs={"preserve_thinking": false} --kv-cache-dtype=fp8_e4m3 --tensor-parallel-size=1 --pipeline-parallel-size=1 --gpu-memory-utilization=0.948 --trust-remote-code --attention-backend=TRITON_ATTN --no-use-tqdm-on-load --host=0.0.0.0 --port=5001 --limit-mm-per-prompt={"image":0,"video":0} --speculative-config={"method":"mtp","num_speculative_tokens":3}
============================================================
[launcher] tee stdout -> C:\Users\nordk\Desktop\qwen3.6-windows-server\logs\vllm_server.5001.log (also streaming to this terminal)
[launcher] runtime manifest -> C:\Users\nordk\Desktop\qwen3.6-windows-server\logs\runtime\5001.json
WARNING 05-27 12:15:32 [cuda.py:801] Detected different devices in the system: NVIDIA GeForce RTX 5070, NVIDIA GeForce RTX 3090. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior.
(APIServer pid=23132) INFO 05-27 12:15:38 [utils.py:299]

(APIServer pid=23132) INFO 05-27 12:15:38 [utils.py:299]        █     █     █▄   ▄█

(APIServer pid=23132) INFO 05-27 12:15:38 [utils.py:299]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.20.0

(APIServer pid=23132) INFO 05-27 12:15:38 [utils.py:299]   █▄█▀ █     █     █     █  model   C:\Users\nordk\Desktop\qwen3.6-windows-server\models\Qwen3.6-27B-int4-AutoRound

(APIServer pid=23132) INFO 05-27 12:15:38 [utils.py:299]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀

(APIServer pid=23132) INFO 05-27 12:15:38 [utils.py:299]
(APIServer pid=23132) INFO 05-27 12:15:38 [utils.py:233] non-default args: {'model_tag': 'C:\\Users\\nordk\\Desktop\\qwen3.6-windows-server\\models\\Qwen3.6-27B-int4-AutoRound', 'chat_template': 'C:\\Users\\nordk\\Desktop\\qwen3.6-windows-server\\templates\\qwen3.5-enhanced.jinja', 'default_chat_template_kwargs': {'preserve_thinking': False}, 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'host': '0.0.0.0', 'port': 5001, 'model': 'C:\\Users\\nordk\\Desktop\\qwen3.6-windows-server\\models\\Qwen3.6-27B-int4-AutoRound', 'trust_remote_code': True, 'max_model_len': 127000, 'quantization': 'auto-round', 'served_model_name': ['qwen3.6-27b-autoround'], 'use_tqdm_on_load': False, 'attention_backend': 'TRITON_ATTN', 'reasoning_parser': 'qwen3', 'block_size': 32, 'gpu_memory_utilization': 0.948, 'kv_cache_dtype': 'fp8_e4m3', 'enable_prefix_caching': True, 'limit_mm_per_prompt': {'image': 0, 'video': 0}, 'max_num_batched_tokens': 4128, 'max_num_seqs': 1, 'enable_chunked_prefill': True, 'speculative_config': {'method': 'mtp', 'num_speculative_tokens': 3}}
(APIServer pid=23132) INFO 05-27 12:15:38 [system_utils.py:284] Windows detected, skipping ulimit adjustment.
(APIServer pid=23132) WARNING 05-27 12:15:38 [envs.py:1821] Unknown vLLM environment variable detected: VLLM_ATTENTION_BACKEND
(APIServer pid=23132) WARNING 05-27 12:15:38 [envs.py:1821] Unknown vLLM environment variable detected: VLLM_MODEL_DIR
(APIServer pid=23132) WARNING 05-27 12:15:38 [envs.py:1821] Unknown vLLM environment variable detected: VLLM_SLEEP_WHEN_IDLE
(APIServer pid=23132) INFO 05-27 12:15:38 [model.py:563] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=23132) INFO 05-27 12:15:38 [model.py:1692] Using max model len 127000
(APIServer pid=23132) INFO 05-27 12:15:40 [nixl_utils.py:20] Setting UCX_RCACHE_MAX_UNRELEASED to '1024' to avoid a rare memory leak in UCX when using NIXL.
(APIServer pid=23132) WARNING 05-27 12:15:40 [nixl_utils.py:34] NIXL is not available
(APIServer pid=23132) WARNING 05-27 12:15:40 [nixl_utils.py:44] NIXL agent config is not available
(APIServer pid=23132) INFO 05-27 12:15:40 [cache.py:261] Using fp8_e4m3 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=23132) INFO 05-27 12:15:40 [model.py:563] Resolved architecture: Qwen3_5MTP
(APIServer pid=23132) INFO 05-27 12:15:40 [model.py:1692] Using max model len 262144
(APIServer pid=23132) WARNING 05-27 12:15:40 [speculative.py:654] Enabling num_speculative_tokens > 1 will run multiple times of forward on same MTP layer,which may result in lower acceptance rate
(APIServer pid=23132) INFO 05-27 12:15:40 [scheduler.py:239] Chunked prefill is enabled with max_num_batched_tokens=4128.
(APIServer pid=23132) WARNING 05-27 12:15:40 [config.py:367] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=23132) INFO 05-27 12:15:40 [config.py:387] Warning: Prefix caching in Mamba cache 'align' mode is currently enabled. Its support for Mamba layers is experimental. Please report any issues you may observe.
(APIServer pid=23132) INFO 05-27 12:15:40 [vllm.py:840] Asynchronous scheduling is enabled.
(APIServer pid=23132) INFO 05-27 12:15:40 [kernel.py:205] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'])
(APIServer pid=23132) WARNING 05-27 12:15:40 [vllm.py:1402] max_num_scheduled_tokens is set to 4128 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
(APIServer pid=23132) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(APIServer pid=23132) INFO 05-27 12:15:42 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
WARNING 05-27 12:15:52 [cuda.py:801] Detected different devices in the system: NVIDIA GeForce RTX 5070, NVIDIA GeForce RTX 3090. Please make sure to set `CUDA_DEVICE_ORDER=PCI_BUS_ID` to avoid unexpected behavior.
(EngineCore pid=7840) INFO 05-27 12:15:55 [core.py:109] Initializing a V1 LLM engine (v0.20.0) with config: model='C:\\Users\\nordk\\Desktop\\qwen3.6-windows-server\\models\\Qwen3.6-27B-int4-AutoRound', speculative_config=SpeculativeConfig(method='mtp', model='C:\\Users\\nordk\\Desktop\\qwen3.6-windows-server\\models\\Qwen3.6-27B-int4-AutoRound', num_spec_tokens=3), tokenizer='C:\\Users\\nordk\\Desktop\\qwen3.6-windows-server\\models\\Qwen3.6-27B-int4-AutoRound', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=127000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=inc, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8_e4m3, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3.6-27b-autoround, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [4128], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 8, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto')
(EngineCore pid=7840) WARNING 05-27 12:15:55 [nixl_utils.py:34] NIXL is not available
(EngineCore pid=7840) WARNING 05-27 12:15:55 [nixl_utils.py:44] NIXL agent config is not available
(EngineCore pid=7840) [transformers] `Qwen2VLImageProcessorFast` is deprecated. The `Fast` suffix for image processors has been removed; use `Qwen2VLImageProcessor` instead.
(EngineCore pid=7840) INFO 05-27 12:15:58 [registry.py:126] All limits of multimodal modalities supported by the model are set to 0, running in text-only mode.
(EngineCore pid=7840) INFO 05-27 12:15:58 [parallel_state.py:1440] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.1.72:52151 backend=gloo
[W527 12:15:58.000000000 socket.cpp:764] [c10d] The client socket has failed to connect to [NorD-PC]:52151 (system error: 10049 - ��������� ����� ��� ������ ��������� �������.).
(EngineCore pid=7840) INFO 05-27 12:15:58 [parallel_state.py:1753] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136] EngineCore failed to start.

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136] Traceback (most recent call last):

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\engine\core.py", line 1110, in run_engine_core

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\tracing\otel.py", line 178, in sync_wrapper

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     return func(*args, **kwargs)

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\engine\core.py", line 876, in __init__

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     super().__init__(

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\engine\core.py", line 118, in __init__

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     self.model_executor = executor_class(vllm_config)

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\tracing\otel.py", line 178, in sync_wrapper

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     return func(*args, **kwargs)

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\executor\abstract.py", line 109, in __init__

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     self._init_executor()

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\executor\uniproc_executor.py", line 47, in _init_executor

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     self.driver_worker.init_device()

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\worker\worker_base.py", line 317, in init_device

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     self.worker.init_device()  # type: ignore

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     ^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\tracing\otel.py", line 178, in sync_wrapper

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     return func(*args, **kwargs)

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\worker\gpu_worker.py", line 284, in init_device

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     self.requested_memory = request_memory(init_snapshot, self.cache_config)

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\worker\utils.py", line 413, in request_memory

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136]     raise ValueError(

(EngineCore pid=7840) ERROR 05-27 12:15:58 [core.py:1136] ValueError: Free memory on device cuda:0 (10.78/11.94 GiB) on startup is less than desired GPU memory utilization (0.948, 11.32 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.
(EngineCore pid=7840) Process EngineCore:
(EngineCore pid=7840) Traceback (most recent call last):
(EngineCore pid=7840)   File "multiprocessing\process.py", line 314, in _bootstrap
(EngineCore pid=7840)   File "multiprocessing\process.py", line 108, in run
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\engine\core.py", line 1140, in run_engine_core
(EngineCore pid=7840)     raise e
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\engine\core.py", line 1110, in run_engine_core
(EngineCore pid=7840)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=7840)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\tracing\otel.py", line 178, in sync_wrapper
(EngineCore pid=7840)     return func(*args, **kwargs)
(EngineCore pid=7840)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\engine\core.py", line 876, in __init__
(EngineCore pid=7840)     super().__init__(
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\engine\core.py", line 118, in __init__
(EngineCore pid=7840)     self.model_executor = executor_class(vllm_config)
(EngineCore pid=7840)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\tracing\otel.py", line 178, in sync_wrapper
(EngineCore pid=7840)     return func(*args, **kwargs)
(EngineCore pid=7840)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\executor\abstract.py", line 109, in __init__
(EngineCore pid=7840)     self._init_executor()
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\executor\uniproc_executor.py", line 47, in _init_executor
(EngineCore pid=7840)     self.driver_worker.init_device()
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\worker\worker_base.py", line 317, in init_device
(EngineCore pid=7840)     self.worker.init_device()  # type: ignore
(EngineCore pid=7840)     ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\tracing\otel.py", line 178, in sync_wrapper
(EngineCore pid=7840)     return func(*args, **kwargs)
(EngineCore pid=7840)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\worker\gpu_worker.py", line 284, in init_device
(EngineCore pid=7840)     self.requested_memory = request_memory(init_snapshot, self.cache_config)
(EngineCore pid=7840)                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=7840)   File "C:\Users\nordk\Desktop\qwen3.6-windows-server\python\Lib\site-packages\vllm\v1\worker\utils.py", line 413, in request_memory
(EngineCore pid=7840)     raise ValueError(
(EngineCore pid=7840) ValueError: Free memory on device cuda:0 (10.78/11.94 GiB) on startup is less than desired GPU memory utilization (0.948, 11.32 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.

Verifier output

python windows_tools\verify_install.py --venv .\venv

[Errno 2] No such file or directory

What you've already tried

Change gpu in config

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions