[GGUF] Serialize Generated OV Model for Faster LLMPipeline Init

[sammysun0711]

Details:
This PR aim to cache generated OV model from GGUF model in disk for faster subsequent pipe initialization w/ OpenVINO model cache.

• Serialize generated OV model from GGUF model w/ GGUF Reader
• Checked if OV model exists in same folder of GGUF model, skip create GGUF model w/ GGUF Reader

Expected behavior:

• First run w/ GGUF model:

  • build/samples/cpp/text_generation/greedy_causal_lm gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf "Who are you?" gguf_models/qwen2.5_openvino_tokenizer

  • Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
    Loading and unpacking model done. Time: 245ms
    Start generating OpenVINO model...
    Save generated OpenVINO model to: gguf_models/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml done. Time: 423 ms
    Model generation done. Time: 968ms
    I am Qwen, a large language model created by Alibaba Cloud. I am a language model designed to assist users in generating human-like text, such as writing articles, stories, and even writing books. I am trained on a vast corpus of text data, including books, articles, and other written works. I am also trained on a large corpus of human language data, including written and spoken language. I am designed to provide information and insights to users, and to assist them in their tasks and

• 2nd run w/ GGUF model:

  • build/samples/cpp/text_generation/greedy_causal_lm gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf "Who are you?" gguf_models/qwen2.5_openvino_tokenizer

  • Found generated OpenVINO model: gguf_models/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml, skip creating from GGUF model.
    I am Qwen, a large language model created by Alibaba Cloud. I am a language model designed to assist users in generating human-like text, such as writing articles, stories, and even writing books. I am trained on a vast corpus of text data, including books, articles, and other written works. I am also trained on a large corpus of human language data, including written and spoken language. I am designed to provide information and insights to users, and to assist them in their tasks and

[Copilot]

Pull Request Overview

This PR enhances the model initialization performance by caching generated OpenVINO models on disk and reusing them for subsequent runs. Key changes include:

• In src/cpp/src/utils.cpp, adding a check for an existing cached OpenVINO model based on the GGUF model location.
• In src/cpp/src/gguf_utils/gguf_modeling.cpp, introducing a new function to serialize and save the generated OpenVINO model, and invoking it during model creation.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
src/cpp/src/utils.cpp	Added logic to reuse a cached OpenVINO model if it exists.
src/cpp/src/gguf_utils/gguf_modeling.cpp	Added serialization function and integrated it into the model creation flow.

[as-suvorov]

@sammysun0711
Proposal looks like implicit model caching. OpenVINO model is being implicitly serialized on disk. And if gguf model has changed it seems outdated serialized OpenVINO model will be loaded.
Can we reuse OpenVINO cache_dir property for such scenario?

Form the logs I see gguf model load time is 245ms. Could you please also provide loading time for OpenVINO serialized model?

[sammysun0711]

Hi @as-suvorov, thanks for your suggestion.

Can we reuse OpenVINO cache_dir property for such scenario?

Do you means if cache_dir property set in property, then we can enable save generated OV model in cache_dir for re-use? I think it is a good proposal that user can control from application whether to serialize OV model in disk explicitly.

Could you please also provide loading time for OpenVINO serialized model?

Sure, I will add loading time for OpenVINO serialized model

[as-suvorov]

@sammysun0711 yes, correct. I guess also need to check if and how OpenVINO invalidates cached model and implement same approach for gguf format.

[sammysun0711]

• Add support to save OV model based on ov::cache_dir properties explicit
• Add time measurement for loading OV model.

1st Run w/o model cache:

Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
Loading and unpacking model done. Time: 202ms
Start generating OpenVINO model...
Model generation done. Time: 292ms

2nd Run w/ model cache + serialize OV model:

Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
Loading and unpacking model done. Time: 189ms
Start generating OpenVINO model...
Save generated OpenVINO model to: model_cache/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml done. Time: 379 ms
Model generation done. Time: 647ms

3rd Run w/ model cache using generated OV model

Found generated OpenVINO model: model_cache/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml, skip creating from GGUF model.
Loading OpenVINO model done. Time: 71ms

check if and how OpenVINO invalidates cached model and implement same approach for gguf format

Current naive method use GGUF file name to check if new OV model need to be regenerated, OV invalidate outdated model cache via hash calculate by compute_hash.hpp, but it seems compute_hash.hpp belong to dev_api, which is not accessible unless openvino.genai static link to openvino. @as-suvorov, may I know if you have an suggestions?

[sammysun0711]

I added initial hash function in test branch, while found that hash function overhead is higher than model construction method.

Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
Loading and unpacking model done. Time: 185ms
Compute GGUF hash with SHA1 done. Time: 856ms
Start generating OpenVINO model...
Save generated OpenVINO model to: model_cache/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml done. Time: 567 ms
Model generation done. Time: 1702ms

Although can try to optimize hash method with parallelism, the hash function will be called for gguf load, which is not idea if we would like to skip gguf model load & unpack process if OV model existed in model cache.

I would like to suggest to serialize OV model in cache_dir when user set cache_dir property explicitly, and add verbose that existed cache OV model will skip GGUF->OV conversion w/o cache invalidate. User may need to clean up model cache and re-generate OV model if gguf model updated.

We can add hash function check after compute_hash.hpp expose as public API for re-use.

[as-suvorov]

I added initial hash function in test branch, while found that hash function overhead is higher than model construction method.

Loading and unpacking model from: gguf_models/qwen2.5-0.5b-instruct-q4_0.gguf
Loading and unpacking model done. Time: 185ms
Compute GGUF hash with SHA1 done. Time: 856ms
Start generating OpenVINO model...
Save generated OpenVINO model to: model_cache/qwen2.5-0.5b-instruct-q4_0_openvino_model.xml done. Time: 567 ms
Model generation done. Time: 1702ms

Although can try to optimize hash method with parallelism, the hash function will be called for gguf load, which is not idea if we would like to skip gguf model load & unpack process if OV model existed in model cache.

I would like to suggest to serialize OV model in cache_dir when user set cache_dir property explicitly, and add verbose that existed cache OV model will skip GGUF->OV conversion w/o cache invalidate. User may need to clean up model cache and re-generate OV model if gguf model updated.

We can add hash function check after compute_hash.hpp expose as public API for re-use.

@Wovchena What do you think?

[rkazants]

why?

[sammysun0711]

Test purpose only, try to figure out why MacOS test failed (segment fault): https://github.com/openvinotoolkit/openvino.genai/actions/runs/15420607456/job/43434100193

It seems macos-13 has smaller memory size: https://docs.github.com/en/actions/using-github-hosted-runners/using-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories

Does it make sense to create a separate test case, instead of check 3 pipeline test (HF/GGUF/OV native) to only 2 pipeline test (GGUF/OV native) to save memory usage?

[rkazants]

@mryzhov, please advice other MacOS runner with more memory

[sammysun0711]

@mryzhov, can we try macos-13-large (30GB memory) for debug purpose first? : https://docs.github.com/en/actions/using-github-hosted-runners/using-larger-runners/running-jobs-on-larger-runners?platform=mac#available-macos-larger-runners

[sammysun0711]

test results shows that not related to insufficient memory, https://github.com/openvinotoolkit/openvino.genai/actions/runs/15436902324/job/43447403549?pr=2218, will revert back to macos-13 and continue to investigation.

[sammysun0711]

@akladiev provides my a local macos machine w/ macos-12 + 64GB memory (during test the memory usage ~30GB + 15GB Swap usage), cannot reproduce the same segment fault issue w/ GGUF related test in test_llm_pipeline.py

I found the same issue in other PR：Bump actions/download-artifact from 4.1.9 to 4.3.0 · openvinotoolkit/openvino.genai@c3fe438, so I think it not introduced by my PR, could you please help to further investigate this issue?

[rkazants]

need review again