cuda out of memory

[Shengqi77]

When I run python inference-image.py --bf16 --score on a 3090 Ti, the following situation occurs. What is the minimum VRAM required?

[Uranusxer]

Our base model of the VisionRewrad-Image is CogVLM2-llama3-chat-19B. So the minimum VRAM required to run inference-image.py with --bf16 --score is 40 GiB.

[weathon]

Is it possible to have quantization