Skip to content

Commit f9dad2a

Browse files
committed
Update README
1 parent 33e61e7 commit f9dad2a

File tree

2 files changed

+4
-4
lines changed

2 files changed

+4
-4
lines changed

README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,12 @@ vec-inf launch Meta-Llama-3.1-8B-Instruct
1616
```
1717
You should see an output like the following:
1818

19-
<img width="400" alt="launch_img" src="https://github.com/user-attachments/assets/9d29947a-2708-4131-9a78-4484d2361da3">
19+
<img width="700" alt="launch_img" src="https://github.com/user-attachments/assets/ab658552-18b2-47e0-bf70-e539c3b898d5">
2020

2121
The model would be launched using the [default parameters](vec_inf/models/models.csv), you can override these values by providing additional parameters, use `--help` to see the full list. You can also launch your own customized model as long as the model architecture is [supported by vLLM](https://docs.vllm.ai/en/stable/models/supported_models.html), and make sure to follow the instructions below:
2222
* Your model weights directory naming convention should follow `$MODEL_FAMILY-$MODEL_VARIANT`.
2323
* Your model weights directory should contain HF format weights.
24-
* The following launch parameters will conform to default value if not specified: `--max-num-seqs`, `--partition`, `--data-type`, `--venv`, `--log-dir`, `--model-weights-parent-dir`, `--pipeline-parallelism`. All other launch parameters need to be specified for custom models.
24+
* The following launch parameters will conform to default value if not specified: `--max-num-seqs`, `--partition`, `--data-type`, `--venv`, `--log-dir`, `--model-weights-parent-dir`, `--pipeline-parallelism`, `--enforce-eager`. All other launch parameters need to be specified for custom models.
2525
* Example for setting the model weights parent directory: `--model-weights-parent-dir /h/user_name/my_weights`.
2626
* For other model launch parameters you can reference the default values for similar models using the [`list` command ](#list-command).
2727

@@ -68,7 +68,7 @@ You call view the full list of available models by running the `list` command:
6868
```bash
6969
vec-inf list
7070
```
71-
<img width="900" alt="list_img" src="https://github.com/user-attachments/assets/ee53cc36-0681-40e6-ba72-17208df2027c">
71+
<img width="900" alt="list_img" src="https://github.com/user-attachments/assets/7cb2b2ac-d30c-48a8-b773-f648c27d9de2">
7272

7373
You can also view the default setup for a specific supported model by providing the model name, for example `Meta-Llama-3.1-70B-Instruct`:
7474
```bash

vec_inf/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# `vec-inf` Commands
22

33
* `launch`: Specify a model family and other optional parameters to launch an OpenAI compatible inference server, `--json-mode` supported. Check [`here`](./models/README.md) for complete list of available options.
4-
* `list`: List all available model names, `--json-mode` supported.
4+
* `list`: List all available model names, or append a supported model name to view the default configuration, `--json-mode` supported.
55
* `metrics`: Streams performance metrics to the console.
66
* `status`: Check the model status by providing its Slurm job ID, `--json-mode` supported.
77
* `shutdown`: Shutdown a model by providing its Slurm job ID.

0 commit comments

Comments
 (0)