Add Spec dec Bench example

[IzzyPutterman]

What does this PR do?

Type of change: New feature

Overview: Specdec bench example

Usage

# Add a code snippet demonstrating how to use this

Testing

Before your PR is "Ready for review"

• Make sure you read and follow Contributor guidelines and your commits are signed.
• Is this change backward compatible?: Yes
• Did you write any new necessary tests?: No
• Did you add or update any necessary documentation?: Yes
• Did you update Changelog?: Yes

Additional Information

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.52%. Comparing base (8cf516e) to head (17f968e).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #474      +/-   ##
==========================================
+ Coverage   73.46%   73.52%   +0.06%     
==========================================
  Files         180      181       +1     
  Lines       18161    18207      +46     
==========================================
+ Hits        13342    13387      +45     
- Misses       4819     4820       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kevalmorabia97]

Need to mention this new example in https://github.com/NVIDIA/TensorRT-Model-Optimizer/blob/main/CHANGELOG.rst under 0.40 release section (new one)

[kevalmorabia97]

Missing requirements.txt and README.md in examples/specdec_bench

[IzzyPutterman]

Missing requirements.txt and README.md in examples/specdec_bench

I added a section in the readme for how to install. Creating a requirements would get nasty as it techincally supports vLLM, SGLang, and TRTLLM. Simpler to say run in an env which already has one of these installed.

[h-guo18]

Questions: Is there a way to get per_user_tps and per_gpu_tps using specbench?

[IzzyPutterman]

The Generation TPS is per_user_tps technically. I can rename to add a Request prefix to these.

[h-guo18]

Thanks. Is per_gpu_tps feasible to implement in specbench? It would help to plot figures like this, where we can understand the serving performance directly:

[IzzyPutterman]

Updated

[h-guo18]

A high level question: TRTLLM-bench seems to have the same metrics available. Vllm also have a similar vllm bench: link.

Is there any difference between the result from spechbench and trtllm-bench, or shall we reuse existing functionalities from trtllm-bench?

[IzzyPutterman]

A high level question: TRTLLM-bench seems to have the same metrics available. Vllm also have a similar vllm bench: link.

Is there any difference between the result from spechbench and trtllm-bench, or shall we reuse existing functionalities from trtllm-bench?

Great question! Both of those benchmarks try to do the same thing, however they are not unified. With this one you can be guaranteed to send the exact same tokens to the engine in all cases, and have it respect the same chat template and tokenizer. trtllm-bench tends to ignore_eos (which is bad for specdec) and requires the input to already be tokenized (a big source of error depending on how you actually do it).
Also this provides easier ways to get more advanced metrics like how AR/AL changes over the time of a request.

[kevalmorabia97]

Lets add non-deployment related requirements.txt for dependencies not part of modelopt (setup.py) like tiktoken and maybe others

[kevalmorabia97]

Fine since all the 3 base docker images already have everything installed