Documentation regarding the DS format to be fed to GuideLLM #133
Comments
Did you specify |
|
For the other two issues please add more complete stack traces. |
|
Hey @SharonGil, for the second-to-last issue, it looks like the --data was set to HF_MODEL_ID, not sure if this was a typo for the issue or not. But, --data only supports the HF dataset ids. If it is a typo for the issue, a stack trace / example command would be great. For the last one, we pushed a fix for these on the latest main, and we'll be cutting a v0.2.1 release ideally tomorrow to push them up to PyPI. If you install from source with the main branch, that last example should work correctly now. |
|
@sjmonson I tried both with and without the --model, since it was auto-detected from my VLLm (had only mistarl there). |
|
@markurtz Thanks for noticing, yes it was a typo in the issue. |
|
Also, is it correct that currently Llama3 and Mistral tokenizers are not yet supported ? |
Those should be fully supported -- anything that works through AutoTokenizer / AutoProcessor with HuggingFace will work here. If the name of the model on the server doesn't match the HuggingFace ID, though, then it's not possible for us to automatically look that up from HF. In that case, you'll need to use the |
|
I see. Is there an option to upload few more run examples like you did with the synthetic data for other DS configurations? |
|
@SharonGil yes, let me see what I can put together quickly this afternoon, run through some tests on, and then push up in a PR. Will reference in here and tag you. If you need something more immediate than that, let me know the specific use case you're looking at and can get you something over |
|
@markurtz Thanks a lot I appreciate it. No need to hurry, I can wait till your PR. Thanks again. |
|
@SharonGil take a look through PR #137 and see if there's anything more needed there |
Fixes #133 --------- Co-authored-by: Copilot <[email protected]>
There is a need in writing down examples for all DSs formats that can be fed as an input to the GuideLLM benchmark.
Tried using the example in the current README -
guidellm benchmark --target "http://${IP}:${PORT}" --rate-type sweep --max-seconds 30 --data "prompt_tokens=256,output_tokens=128"-Got an error for autoTokenizer not known. After debugging a little thought that it might be that tokenizers for Mistarl and Llama3 aren't supported yet (used VLLM instances with those loaded), and the error occurs since processor is required for synthetic data bencmarking.
Then tried using a HF model ID with -
guidellm benchmark --target "http://${IP}:${PORT}" --rate-type synchronous --max-seconds 30 --data ${HF_DS_ID} --data-args '{"prompt_column": "prompt"}'and the benchmark ran but got an error in openAIHTTPBackend for every request, which made me think that the data format I'm sending isn't correct.then tried to used a local JSON file as a DS and ran with -
guidellm benchmark --target "http://${IP}:${PORT}" --rate-type synchronous --max-seconds 30 --data "prompts.json"ad got a
TypeError: 'PosixPath' object is not iterableerrorWould appreciate help in running GuideLLM benchmarks with different DSs configurations.
The text was updated successfully, but these errors were encountered: