Skip to content

ft支持w4a8, w4a8 moe, fp8 量化（仅针对SM90 卡子） - #706

Open
moguizhizi wants to merge 46 commits into
ztxz16:masterfrom
moguizhizi:master
Open

ft支持w4a8, w4a8 moe, fp8 量化（仅针对SM90 卡子）#706
moguizhizi wants to merge 46 commits into
ztxz16:masterfrom
moguizhizi:master

Conversation

@moguizhizi

@moguizhizi moguizhizi commented Jul 30, 2026

Copy link
Copy Markdown

ft支持 w4a8, w4a8 moe 算子， fp8 量化算子

moguizhizi added 30 commits July 29, 2026 13:56
@moguizhizi
补充W4A8独立入口
81febf6
@moguizhizi
整理W4A8依赖
c60dad0
@moguizhizi
补充W4A8权重缓存
42539b5
@moguizhizi
补充W4A8激活量化
abcf9f1
@moguizhizi
补充W4A8输出语义
30b423a
@moguizhizi
补充W4A8验证日志
afdcb0a
@moguizhizi
补充W4A8执行开关
fa2cf4d
@moguizhizi
接入W4A8 GEMM调用
9a823cf
@moguizhizi
补充W4A8 CUDA测试
a9b9502
@moguizhizi
修复W4A8 epilogue编译冲突
3a5e8e6
@moguizhizi
修正W4A8运行时架构判断
54084ae
@moguizhizi
修正W4A8尺度布局
36d128a
@moguizhizi
修正W4A8权重打包布局
447f691
@moguizhizi
调整W4A8测试容差
61f496c
@moguizhizi
接入W4A8压缩权重加载
5225dc2
@moguizhizi
完善W4A8独立权重语义
d1c95fe
@moguizhizi
接入W4A8权重转换缓存
0fa0654
@moguizhizi
完善W4A8缓存生命周期和正式分发
aaea51c
@moguizhizi
明确W4A8非SM90处理策略
52e93bc
@moguizhizi
补充W4A8真实权重加载测试
cf4d474
@moguizhizi
补充W4A8向量化依赖
5230b9e
@moguizhizi
接入原生W4A8算子路径
d20bb25
@moguizhizi
补充W4A8线性算子测试
3e7bfff
@moguizhizi
修复W4A8算子测试编译
30c60dd
@moguizhizi
优化W4A8激活动态量化
4f9082a
@moguizhizi
补充W4A8整模性能配置
e3e44b0
@moguizhizi
修复W4A8头文件搜索路径
ddbe989
@moguizhizi
修复forward_check模型加载
a1f0cb8
@moguizhizi
兼容量化模型forward_check
dbc7eab
@moguizhizi
稳定forward_check参考端Attention
3806099
@moguizhizi

moguizhizi commented Jul 30, 2026

Copy link
Copy Markdown
Author

ENV INFO:
Collecting environment information...
PyTorch version: 2.11.0+cu130
Is debug build: False
CUDA used to build PyTorch: 13.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.12.3 | packaged by Anaconda, Inc. | (main, May 6 2024, 19:46:43) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-161-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.8.93
CUDA_MODULE_LOADING set to:
GPU models and configuration: GPU 0: NVIDIA H800 PCIe
Nvidia driver version: 580.82.07
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.8.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.8.0
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Caching allocator config: N/A

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 176
On-line CPU(s) list: 0-175
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8458P
CPU family: 6
Model: 143
Thread(s) per core: 2
Core(s) per socket: 44
Socket(s): 2
Stepping: 8
BogoMIPS: 5400.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cat_l2 cdp_l3 invpcid_single intel_ppin cdp_l2 ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect avx_vnni avx512_bf16 wbnoinvd dtherm arat pln pts avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid bus_lock_detect cldemote movdiri movdir64b enqcmd fsrm md_clear serialize tsxldtrk pconfig arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities ibpb_exit_to_user
Virtualization: VT-x
L1d cache: 4.1 MiB (88 instances)
L1i cache: 2.8 MiB (88 instances)
L2 cache: 176 MiB (88 instances)
L3 cache: 165 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-43,88-131
NUMA node1 CPU(s): 44-87,132-175
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Mitigation; IBPB before exit to userspace

Versions of relevant libraries:
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cublas-cu12==12.8.3.14
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-cupti-cu12==12.8.57
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-nvrtc-cu12==12.8.61
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cuda-runtime-cu12==12.8.57
[pip3] nvidia-cudnn-cu12==9.7.1.26
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.26.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufft-cu12==11.3.3.41
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-curand-cu12==10.3.9.55
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusolver-cu12==11.7.2.55
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparse-cu12==12.5.7.53
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvjitlink-cu12==12.8.61
[pip3] nvidia-nvtx==13.0.85
[pip3] nvidia-nvtx-cu12==12.8.55
[pip3] nvtx==0.2.15
[pip3] tokenspeed-triton==3.8.10.post20260721
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torchaudio==2.11.0
[pip3] torchcodec==0.15.0
[pip3] torchvision==0.26.0
[pip3] triton==3.6.0
[conda] numpy 2.3.5 pypi_0 pypi
[conda] nvidia-cublas 13.1.0.3 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.8.3.14 pypi_0 pypi
[conda] nvidia-cuda-cupti 13.0.85 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.8.57 pypi_0 pypi
[conda] nvidia-cuda-nvrtc 13.0.88 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.8.61 pypi_0 pypi
[conda] nvidia-cuda-runtime 13.0.96 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.8.57 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.7.1.26 pypi_0 pypi
[conda] nvidia-cudnn-cu13 9.19.0.56 pypi_0 pypi
[conda] nvidia-cudnn-frontend 1.26.0 pypi_0 pypi
[conda] nvidia-cufft 12.0.0.61 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.3.3.41 pypi_0 pypi
[conda] nvidia-curand 10.4.0.35 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.9.55 pypi_0 pypi
[conda] nvidia-cusolver 12.0.4.66 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.7.2.55 pypi_0 pypi
[conda] nvidia-cusparse 12.6.3.3 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.5.7.53 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.6.3 pypi_0 pypi
[conda] nvidia-cusparselt-cu13 0.8.0 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.26.2 pypi_0 pypi
[conda] nvidia-nccl-cu13 2.28.9 pypi_0 pypi
[conda] nvidia-nvjitlink 13.0.88 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.8.61 pypi_0 pypi
[conda] nvidia-nvtx 13.0.85 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.8.55 pypi_0 pypi
[conda] nvtx 0.2.15 pypi_0 pypi
[conda] tokenspeed-triton 3.8.10.post20260721 pypi_0 pypi
[conda] torch 2.11.0+cu130 pypi_0 pypi
[conda] torch-c-dlpack-ext 0.1.5 pypi_0 pypi
[conda] torchaudio 2.11.0 pypi_0 pypi
[conda] torchcodec 0.15.0 pypi_0 pypi
[conda] torchvision 0.26.0 pypi_0 pypi
[conda] triton 3.6.0 pypi_0 pypi

==============================
FastLLM Info

FastLLM package : ftllm 0.1.7.1
FastLLM package location : /root/miniconda3/lib/python3.12/site-packages/ftllm/init.py
FastLLM git revision : Could not collect
FastLLM git status : Could not collect
FastLLM build flags : {"USE_CUDA": true, "USE_IVCOREX": false, "USE_NUMAS": true, "USE_ROCM": false, "USE_TFACC": false, "USE_TOPS": false}
Python executable : /root/miniconda3/bin/python
Platform : Linux-5.15.0-161-generic-x86_64-with-glibc2.35
GCC version : gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version : Could not collect
CMake version : cmake version 3.22.1
NVCC version : nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0

==============================
GPU Topology

�[4mGPU0 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID�[0m
GPU0 X SYS SYS NODE NODE 44-87,132-175 1 N/A
NIC0 SYS X PIX SYS SYS
NIC1 SYS PIX X SYS SYS
NIC2 NODE SYS SYS X PIX
NIC3 NODE SYS SYS PIX X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3

==============================
Relevant Python Packages

cuda-bindings==13.3.1
cuda-core==1.0.1
cuda-pathfinder==1.6.0
cuda-python==13.3.1
cuda-tile==1.3.0
cuda-toolkit==13.0.2
flashinfer-cubin==0.6.13
flashinfer-python==0.6.13
ftllm==0.1.7.1
numpy==2.3.5
nvidia-cublas==13.1.0.3
nvidia-cublas-cu12==12.8.3.14
nvidia-cuda-cccl==13.3.3.4.1
nvidia-cuda-crt==13.3.73
nvidia-cuda-cupti==13.0.85
nvidia-cuda-cupti-cu12==12.8.57
nvidia-cuda-nvcc==13.2.86
nvidia-cuda-nvrtc==13.0.88
nvidia-cuda-nvrtc-cu12==12.8.61
nvidia-cuda-runtime==13.0.96
nvidia-cuda-runtime-cu12==12.8.57
nvidia-cuda-tileiras==13.2.86
nvidia-cudnn-cu12==9.7.1.26
nvidia-cudnn-cu13==9.19.0.56
nvidia-cudnn-frontend==1.26.0
nvidia-cufft==12.0.0.61
nvidia-cufft-cu12==11.3.3.41
nvidia-cufile==1.15.1.6
nvidia-cufile-cu12==1.13.0.11
nvidia-curand==10.4.0.35
nvidia-curand-cu12==10.3.9.55
nvidia-cusolver==12.0.4.66
nvidia-cusolver-cu12==11.7.2.55
nvidia-cusparse==12.6.3.3
nvidia-cusparse-cu12==12.5.7.53
nvidia-cusparselt-cu12==0.6.3
nvidia-cusparselt-cu13==0.8.0
nvidia-cutlass-dsl==4.5.2
nvidia-cutlass-dsl-libs-base==4.5.2
nvidia-cutlass-dsl-libs-cu13==4.5.2
nvidia-ml-py==13.610.43
nvidia-nccl-cu12==2.26.2
nvidia-nccl-cu13==2.28.9
nvidia-nvjitlink==13.0.88
nvidia-nvjitlink-cu12==12.8.61
nvidia-nvshmem-cu13==3.4.5
nvidia-nvtx==13.0.85
nvidia-nvtx-cu12==12.8.55
nvidia-nvvm==13.2.86
tokenspeed-triton==3.8.10.post20260721
torch==2.11.0+cu130
torch_c_dlpack_ext==0.1.5
torchaudio==2.11.0
torchcodec==0.15.0
torchvision==0.26.0
transformers==5.13.0.dev0
triton==3.6.0

==============================
Environment Variables

CUDA_HOME=/usr/local/cuda
MKL_NUM_THREADS=20
OMP_NUM_THREADS=20

@moguizhizi moguizhizi changed the title ft支持w4a8, w4a8 moe, fp8 量化 ft支持w4a8, w4a8 moe, fp8 量化（仅针对SM90 卡子） Jul 30, 2026
@moguizhizi

moguizhizi commented Jul 30, 2026

Copy link
Copy Markdown
Author

验证的模型：
czhu-cohere/Qwen3-30B-A3B-quantized.w4a8-v2
Etelis/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ

@moguizhizi

moguizhizi commented Jul 30, 2026

Copy link
Copy Markdown
Author

w4a8 算子功能，性能的测试

CPU Instruction Info: [AVX2: ON] [AVX512F: ON] [AVX512_VNNI: ON] [AVX512_BF16: ON] [AMX: ON]

== op: linear ==
fully connected layer
batch=16 # batch size
in=4096 # input features
out=6144 # output features
weight_type=int4_w4a8 # weight datatype: float32, int4group, int4group32 or int4_w4a8
group_cnt=128 # group size used by int4group quantization
[cuda:0] PASS
accuracy: max_abs_diff=1.953125e-02, max_rel_diff=1.626350e+02
latency: avg_ms=1.3533
throughput: io_speed=9.8484 GB/s, compute_speed=595.1193 GOPS

Summary: PASS

w4a8 forward_check测试

2026-07-30 22:48:37,907 - INFO - Namespace(model='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', tokens=8, hf_device='cuda', flm_dtype='bfloat16', flm_atype='bfloat16', flm_threads=4, flm_device='cuda')
2026-07-30 22:48:37,907 - INFO - 开始测试模型 /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
2026-07-30 22:48:37,907 - INFO - 正在用Transformer读取模型
Load libfastllm_tools.so

Compressing model: 0%| | 0/224 [00:00<?, ?it/s]
Compressing model: 7%|▋ | 16/224 [00:00<00:01, 157.34it/s]
Compressing model: 15%|█▍ | 33/224 [00:00<00:01, 163.65it/s]
Compressing model: 22%|██▏ | 50/224 [00:00<00:01, 165.71it/s]
Compressing model: 30%|██▉ | 67/224 [00:00<00:00, 166.31it/s]
Compressing model: 38%|███▊ | 84/224 [00:00<00:00, 167.07it/s]
Compressing model: 45%|████▌ | 101/224 [00:00<00:00, 167.36it/s]
Compressing model: 53%|█████▎ | 118/224 [00:00<00:00, 167.79it/s]
Compressing model: 60%|██████ | 135/224 [00:00<00:00, 168.04it/s]
Compressing model: 68%|██████▊ | 152/224 [00:00<00:00, 168.12it/s]
Compressing model: 75%|███████▌ | 169/224 [00:01<00:00, 168.08it/s]
Compressing model: 83%|████████▎ | 186/224 [00:01<00:00, 168.15it/s]
Compressing model: 91%|█████████ | 203/224 [00:01<00:00, 168.18it/s]
Compressing model: 98%|█████████▊| 220/224 [00:01<00:00, 168.16it/s]
Compressing model: 100%|██████████| 224/224 [00:01<00:00, 169.46it/s]

Loading weights: 0%| | 0/739 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 739/739 [00:00<00:00, 12299.47it/s]
2026-07-30 22:48:42,559 - INFO - 读取成功
2026-07-30 22:48:42,560 - INFO - 正在用Fastllm读取模型
CPU Instruction Info: [AVX2: ON] [AVX512F: ON] [AVX512_VNNI: ON] [AVX512_BF16: ON] [AMX: ON]
[Fastllm] compressed-tensors W4A8: grouped 224 Linear weights.
Load 0
Load 0
Load 1
Load 1
Load 1
Load 2
Load 2
Load 2
Load 3
Load 3
Load 3
Load 4
Load 4
Load 4
Load 5
Load 5
Load 5
Load 6
Load 6
Load 6
Load 7
Load 7
Load 7
Load 8
Load 8
Load 8
Load 9
Load 9
Load 9
Load 10
Load 10
Load 10
Load 11
Load 11
Load 12
Load 12
Load 12
Load 13
Load 13
Load 13
Load 14
Load 14
Load 14
Load 15
Load 15
Load 15
Load 16
Load 16
Load 16
Load 17
Load 17
Load 17
Load 18
Load 18
Load 18
Load 19
Load 19
Load 19
Load 20
Load 20
Load 20
Load 21
Load 21
Load 21
Load 22
Load 22
Load 23
Load 23
Load 23
Load 24
Load 24
Load 24
Load 25
Load 25
Load 25
Load 26
Load 26
Load 26
Load 27
Load 27
Load 27
Load 28
Load 28
Load 28
Load 29
Load 29
Load 29
Load 30
Load 30
Load 30
Load 31
Load 31
Load 31
Load 32
Load 32
Load 32
Load 33
Load 33
Load 34
Load 34
Load 34
Load 35
Load 35
Load 35
Load 36
Load 36
Load 36
Load 37
Load 37
Load 37
Load 38
Load 38
Load 38
Load 39
Load 39
Load 39
Load 40
Load 40
Load 40
Load 41
Load 41
Load 41
Load 42
Load 42
Load 42
Load 43
Load 43
Load 43
Load 44
Load 44
Load 45
Load 45
Load 45
Load 46
Load 46
Load 46
Load 47
Load 47
Load 47
Load 48
Load 48
Load 48
Load 49
Load 49
Load 49
Load 50
Load 50
Load 50
Load 51
Load 51
Load 51
Load 52
Load 52
Load 52
Load 53
Load 53
Load 53
Load 54
Load 54
Load 54
Load 55
Load 55
Load 56
Load 56
Load 56
Load 57
Load 57
Load 57
Load 58
Load 58
Load 58
Load 59
Load 59
Load 59
Load 60
Load 60
Load 60
Load 61
Load 61
Load 61
Load 62
Load 62
Load 62
Load 63
Load 63
Load 63
Load 64
Load 64
Load 64
Load 65
Load 65
Load 65
Load 66
Load 66
Load 67
Load 67
Load 67
Load 68
Load 68
Load 68
Load 69
Load 69
Load 69
Load 70
Load 70
Load 70
Load 71
Load 71
Load 71
Load 72
Load 72
Load 72
Load 73
Load 73
Load 73
Load 74
Load 74
Load 74
Load 75
Load 75
Load 75
Load 76
Load 76
Load 76
Load 77
Load 77
Load 78
Load 78
Load 78
Load 79
Load 79
Load 79
Load 80
Load 80
Load 80
Load 81
Load 81
Load 81
Load 82
Load 82
Load 82
Load 83
Load 83
Load 83
Load 84
Load 84
Load 84
Load 85
Load 85
Load 85
Load 86
Load 86
Load 86
Load 87
Load 87
Load 87
Load 88
Load 88
Load 89
Load 89
Load 89
Load 90
Load 90
Load 90
Load 91
Load 91
Load 91
Load 92
Load 92
Load 92
Load 93
Load 93
Load 93
Load 94
Load 94
Load 94
Load 95
Load 95
Load 95
Load 96
Load 96
Load 96
Load 97
Load 97
Load 97
Load 98
Load 98
Load 98
Load 99
Load 99
Load 100
Loading 0
Loading 0
Loading 1
Loading 1
Loading 1
Loading 2
Loading 2
Loading 2
Loading 3
Loading 3
Loading 3
Loading 4
Loading 4
Loading 4
Loading 5
Loading 5
Loading 5
Loading 6
Loading 6
Loading 6
Loading 7
Loading 7
Loading 7
Loading 8
Loading 8
Loading 8
Loading 9
Loading 9
Loading 9
Loading 10
Loading 10
Loading 10
Loading 11
Loading 11
Loading 12
Loading 12
Loading 12
Loading 13
Loading 13
Loading 13
Loading 14
Loading 14
Loading 14
Loading 15
Loading 15
Loading 15
Loading 16
Loading 16
Loading 16
Loading 17
Loading 17
Loading 17
Loading 18
Loading 18
Loading 18
Loading 19
Loading 19
Loading 19
Loading 20
Loading 20
Loading 20
Loading 21
Loading 21
Loading 21
Loading 22
Loading 22
Loading 23
Loading 23
Loading 23
Loading 24
Loading 24
Loading 24
Loading 25
Loading 25
Loading 25
Loading 26
Loading 26
Loading 26
Loading 27
Loading 27
Loading 27
Loading 28
Loading 28
Loading 28
Loading 29
Loading 29
Loading 29
Loading 30
Loading 30
Loading 30
Loading 31
Loading 31
Loading 31
Loading 32
Loading 32
Loading 32
Loading 33
Loading 33
Loading 34
Loading 34
Loading 34
Loading 35
Loading 35
Loading 35
Loading 36
Loading 36
Loading 36
Loading 37
Loading 37
Loading 37
Loading 38
Loading 38
Loading 38
Loading 39
Loading 39
Loading 39
Loading 40
Loading 40
Loading 40
Loading 41
Loading 41
Loading 41
Loading 42
Loading 42
Loading 42
Loading 43
Loading 43
Loading 43
Loading 44
Loading 44
Loading 45
Loading 45
Loading 45
Loading 46
Loading 46
Loading 46
Loading 47
Loading 47
Loading 47
Loading 48
Loading 48
Loading 48
Loading 49
Loading 49
Loading 49
Loading 50
Loading 50
Loading 50
Loading 51
Loading 51
Loading 51
Loading 52
Loading 52
Loading 52
Loading 53
Loading 53
Loading 53
Loading 54
Loading 54
Loading 54
Loading 55
Loading 55
Loading 56
Loading 56
Loading 56
Loading 57
Loading 57
Loading 57
Loading 58
Loading 58
Loading 58
Loading 59
Loading 59
Loading 59
Loading 60
Loading 60
Loading 60
Loading 61
Loading 61
Loading 61
Loading 62
Loading 62
Loading 62
Loading 63
Loading 63
Loading 63
Loading 64
Loading 64
Loading 64
Loading 65
Loading 65
Loading 65
Loading 66
Loading 66
Loading 67
Loading 67
Loading 67
Loading 68
Loading 68
Loading 68
Loading 69
Loading 69
Loading 69
Loading 70
Loading 70
Loading 70
Loading 71
Loading 71
Loading 71
Loading 72
Loading 72
Loading 72
Loading 73
Loading 73
Loading 73
Loading 74
Loading 74
Loading 74
Loading 75
Loading 75
Loading 75
Loading 76
Loading 76
Loading 76
Loading 77
Loading 77
Loading 78
Loading 78
Loading 78
Loading 79
Loading 79
Loading 79
Loading 80
Loading 80
Loading 80
Loading 81
Loading 81
Loading 81
Loading 82
Loading 82
Loading 82
Loading 83
Loading 83
Loading 83
Loading 84
Loading 84
Loading 84
Loading 85
Loading 85
Loading 85
Loading 86
Loading 86
Loading 86
Loading 87
Loading 87
Loading 87
Loading 88
Loading 88
Loading 89
Loading 89
Loading 89
Loading 90
Loading 90
Loading 90
Loading 91
Loading 91
Loading 91
Loading 92
Loading 92
Loading 92
Loading 93
Loading 93
Loading 93
Loading 94
Loading 94
Loading 94
Loading 95
Loading 95
Loading 95
Loading 96
Loading 96
Loading 96
Loading 97
Loading 97
Loading 97
Loading 98
Loading 98
Loading 98
Loading 99
Loading 99
Loading 100
2026-07-30 22:48:47,797 - INFO - 读取成功
2026-07-30 22:48:47,798 - INFO - 使用fastllm进行推理
[Fastllm] default generation config: {'repetition_penalty': 1.0, 'top_p': 0.9, 'top_k': 1, 'temperature': 0.6}

0%| | 0/1 [00:00<?, ?it/s][transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.

100%|██████████| 1/1 [00:07<00:00, 7.42s/it]
100%|██████████| 1/1 [00:07<00:00, 7.42s/it]
2026-07-30 22:48:55,219 - INFO - 释放fastllm模型
2026-07-30 22:48:55,334 - INFO - 使用Transformer进行推理

0%| | 0/1 [00:00<?, ?it/s]

Decompressing model: 0%| | 0/224 [00:00<?, ?it/s]�[A

Decompressing model: 15%|█▌ | 34/224 [00:00<00:00, 333.32it/s]�[A

Decompressing model: 43%|████▎ | 97/224 [00:00<00:00, 505.04it/s]�[A

Decompressing model: 71%|███████▏ | 160/224 [00:00<00:00, 561.36it/s]�[A

Decompressing model: 100%|█████████▉| 223/224 [00:00<00:00, 588.21it/s]�[A
Decompressing model: 100%|██████████| 224/224 [00:00<00:00, 552.02it/s]
[transformers] Both max_new_tokens (=8) and max_length(=4096) seem to have been set. max_new_tokens will take precedence. Please refer to the documentation for more information. (https://huggingface.co/docs/transformers/main/en/main_classes/text_generation)

100%|██████████| 1/1 [00:02<00:00, 2.02s/it]
100%|██████████| 1/1 [00:02<00:00, 2.02s/it]
2026-07-30 22:48:59,071 - INFO - 数据 0 的生成结果相同，结果为 "Beijing, t..."
2026-07-30 22:48:59,096 - INFO - 数据 0 的余弦相似度为0.9936883534586622
2026-07-30 22:48:59,096 - INFO - 平均余弦相似度: 0.9936883534586622

@moguizhizi

moguizhizi commented Jul 30, 2026

Copy link
Copy Markdown
Author
w4a8 prefill性能测试

[1/3] start_case: w4a8-prefill-128
FastLLM Prefill benchmark
case_name: w4a8-prefill-128
repo: /home/vllm_fastllm
base_url: http://127.0.0.1:18080
model_name: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
model_path: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
prefill_length_chars: 128
prompt_chars: 128
max_tokens: 1

2026-07-30 22:51:10,044 13353 server.py[line:305] INFO: Namespace(model=None, path='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', threads=16, low=False, dtype='bfloat16', moe_dtype='', moe_atype='', atype='bfloat16', kv_cache_dtype='auto', cuda_embedding=False, kv_cache_limit='auto', max_batch=1, chunked_prefill_size=-1, device='cuda', tp='', moe_device='numa', moe_device_layers=-1, moe_experts=-1, cache_history='', cache_fast='', enable_thinking='', cuda_shared_expert='true', enable_amx='false', tokens=-1, page_size=-1, prefix_cache='', prefix_cache_snapshot_interval_pages=-1, prefix_cache_snapshot_max_per_request=-1, prefix_cache_snapshot_max_records=-1, gpu_mem_ratio=0.9, cuda_slab=0, mtp=0, speculative_algorithm='', speculative_draft_model_path='', speculative_dspark_block_size=-1, speculative_dspark_confidence_threshold=0.5, triton=False, custom='', lora='', cache_dir='', dtype_config='', ori='', tool_call_parser='auto', chat_template='', model_name='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', host='127.0.0.1', port=18080, api_key='no-key', max_context_length=-1, temperature=None, top_p=None, top_k=None, repeat_penalty=None, think='false', hide_input=True, dev_mode=False, startup_progress='off', config='/tmp/w4a8_prefill_config.json', prompt_repeat=2048, prompt_unit='FastLLM prefill benchmark context block. ', question='请阅读以上上下文，并只回复“测试完成”。', max_tokens=1, startup_timeout=600, request_timeout=3600, warmup_max_tokens=1, skip_warmup=False, name='w4a8-prefill-128', prefill_length=128)
CPU Instruction Info: [AVX2: ON] [AVX512F: ON] [AVX512_VNNI: ON] [AVX512_BF16: ON] [AMX: ON]
[Fastllm] compressed-tensors W4A8: grouped 224 Linear weights.
Load 0
Load 0
Load 1
Load 1
Load 1
Load 2
Load 2
Load 2
Load 3
Load 3
Load 3
Load 4
Load 4
Load 4
Load 5
Load 5
Load 5
Load 6
Load 6
Load 6
Load 7
Load 7
Load 7
Load 8
Load 8
Load 8
Load 9
Load 9
Load 9
Load 10
Load 10
Load 10
Load 11
Load 11
Load 12
Load 12
Load 12
Load 13
Load 13
Load 13
Load 14
Load 14
Load 14
Load 15
Load 15
Load 15
Load 16
Load 16
Load 16
Load 17
Load 17
Load 17
Load 18
Load 18
Load 18
Load 19
Load 19
Load 19
Load 20
Load 20
Load 20
Load 21
Load 21
Load 21
Load 22
Load 22
Load 23
Load 23
Load 23
Load 24
Load 24
Load 24
Load 25
Load 25
Load 25
Load 26
Load 26
Load 26
Load 27
Load 27
Load 27
Load 28
Load 28
Load 28
Load 29
Load 29
Load 29
Load 30
Load 30
Load 30
Load 31
Load 31
Load 31
Load 32
Load 32
Load 32
Load 33
Load 33
Load 34
Load 34
Load 34
Load 35
Load 35
Load 35
Load 36
Load 36
Load 36
Load 37
Load 37
Load 37
Load 38
Load 38
Load 38
Load 39
Load 39
Load 39
Load 40
Load 40
Load 40
Load 41
Load 41
Load 41
Load 42
Load 42
Load 42
Load 43
Load 43
Load 43
Load 44
Load 44
Load 45
Load 45
Load 45
Load 46
Load 46
Load 46
Load 47
Load 47
Load 47
Load 48
Load 48
Load 48
Load 49
Load 49
Load 49
Load 50
Load 50
Load 50
Load 51
Load 51
Load 51
Load 52
Load 52
Load 52
Load 53
Load 53
Load 53
Load 54
Load 54
Load 54
Load 55
Load 55
Load 56
Load 56
Load 56
Load 57
Load 57
Load 57
Load 58
Load 58
Load 58
Load 59
Load 59
Load 59
Load 60
Load 60
Load 60
Load 61
Load 61
Load 61
Load 62
Load 62
Load 62
Load 63
Load 63
Load 63
Load 64
Load 64
Load 64
Load 65
Load 65
Load 65
Load 66
Load 66
Load 67
Load 67
Load 67
Load 68
Load 68
Load 68
Load 69
Load 69
Load 69
Load 70
Load 70
Load 70
Load 71
Load 71
Load 71
Load 72
Load 72
Load 72
Load 73
Load 73
Load 73
Load 74
Load 74
Load 74
Load 75
Load 75
Load 75
Load 76
Load 76
Load 76
Load 77
Load 77
Load 78
Load 78
Load 78
Load 79
Load 79
Load 79
Load 80
Load 80
Load 80
Load 81
Load 81
Load 81
Load 82
Load 82
Load 82
Load 83
Load 83
Load 83
Load 84
Load 84
Load 84
Load 85
Load 85
Load 85
Load 86
Load 86
Load 86
Load 87
Load 87
Load 87
Load 88
Load 88
Load 89
Load 89
Load 89
Load 90
Load 90
Load 90
Load 91
Load 91
Load 91
Load 92
Load 92
Load 92
Load 93
Load 93
Load 93
Load 94
Load 94
Load 94
Load 95
Load 95
Load 95
Load 96
Load 96
Load 96
Load 97
Load 97
Load 97
Load 98
Load 98
Load 98
Load 99
Load 99
Load 100
Loading 0
Loading 0
Loading 1
Loading 1
Loading 1
Loading 2
Loading 2
Loading 2
Loading 3
Loading 3
Loading 3
Loading 4
Loading 4
Loading 4
Loading 5
Loading 5
Loading 5
Loading 6
Loading 6
Loading 6
Loading 7
Loading 7
Loading 7
Loading 8
Loading 8
Loading 8
Loading 9
Loading 9
Loading 9
Loading 10
Loading 10
Loading 10
Loading 11
Loading 11
Loading 12
Loading 12
Loading 12
Loading 13
Loading 13
Loading 13
Loading 14
Loading 14
Loading 14
Loading 15
Loading 15
Loading 15
Loading 16
Loading 16
Loading 16
Loading 17
Loading 17
Loading 17
Loading 18
Loading 18
Loading 18
Loading 19
Loading 19
Loading 19
Loading 20
Loading 20
Loading 20
Loading 21
Loading 21
Loading 21
Loading 22
Loading 22
Loading 23
Loading 23
Loading 23
Loading 24
Loading 24
Loading 24
Loading 25
Loading 25
Loading 25
Loading 26
Loading 26
Loading 26
Loading 27
Loading 27
Loading 27
Loading 28
Loading 28
Loading 28
Loading 29
Loading 29
Loading 29
Loading 30
Loading 30
Loading 30
Loading 31
Loading 31
Loading 31
Loading 32
Loading 32
Loading 32
Loading 33
Loading 33
Loading 34
Loading 34
Loading 34
Loading 35
Loading 35
Loading 35
Loading 36
Loading 36
Loading 36
Loading 37
Loading 37
Loading 37
Loading 38
Loading 38
Loading 38
Loading 39
Loading 39
Loading 39
Loading 40
Loading 40
Loading 40
Loading 41
Loading 41
Loading 41
Loading 42
Loading 42
Loading 42
Loading 43
Loading 43
Loading 43
Loading 44
Loading 44
Loading 45
Loading 45
Loading 45
Loading 46
Loading 46
Loading 46
Loading 47
Loading 47
Loading 47
Loading 48
Loading 48
Loading 48
Loading 49
Loading 49
Loading 49
Loading 50
Loading 50
Loading 50
Loading 51
Loading 51
Loading 51
Loading 52
Loading 52
Loading 52
Loading 53
Loading 53
Loading 53
Loading 54
Loading 54
Loading 54
Loading 55
Loading 55
Loading 56
Loading 56
Loading 56
Loading 57
Loading 57
Loading 57
Loading 58
Loading 58
Loading 58
Loading 59
Loading 59
Loading 59
Loading 60
Loading 60
Loading 60
Loading 61
Loading 61
Loading 61
Loading 62
Loading 62
Loading 62
Loading 63
Loading 63
Loading 63
Loading 64
Loading 64
Loading 64
Loading 65
Loading 65
Loading 65
Loading 66
Loading 66
Loading 67
Loading 67
Loading 67
Loading 68
Loading 68
Loading 68
Loading 69
Loading 69
Loading 69
Loading 70
Loading 70
Loading 70
Loading 71
Loading 71
Loading 71
Loading 72
Loading 72
Loading 72
Loading 73
Loading 73
Loading 73
Loading 74
Loading 74
Loading 74
Loading 75
Loading 75
Loading 75
Loading 76
Loading 76
Loading 76
Loading 77
Loading 77
Loading 78
Loading 78
Loading 78
Loading 79
Loading 79
Loading 79
Loading 80
Loading 80
Loading 80
Loading 81
Loading 81
Loading 81
Loading 82
Loading 82
Loading 82
Loading 83
Loading 83
Loading 83
Loading 84
Loading 84
Loading 84
Loading 85
Loading 85
Loading 85
Loading 86
Loading 86
Loading 86
Loading 87
Loading 87
Loading 87
Loading 88
Loading 88
Loading 89
Loading 89
Loading 89
Loading 90
Loading 90
Loading 90
Loading 91
Loading 91
Loading 91
Loading 92
Loading 92
Loading 92
Loading 93
Loading 93
Loading 93
Loading 94
Loading 94
Loading 94
Loading 95
Loading 95
Loading 95
Loading 96
Loading 96
Loading 96
Loading 97
Loading 97
Loading 97
Loading 98
Loading 98
Loading 98
Loading 99
Loading 99
Loading 100
Load libfastllm_tools.so
Load AutoTokenizer failed. (you can try install transformers)
Try load fastllm tokenizer.
[Fastllm] default generation config: {'repetition_penalty': 1.0, 'top_p': 0.9, 'top_k': 1, 'temperature': 0.6}
2026-07-30 22:51:21,390 13353 server.py[line:321] INFO: Model context window: 8192 tokens per session (model=8192, shared KV cache=None, configured limit=None)
Increased ulimit from 1024 to 65535.
INFO: Started server process [13353]
2026-07-30 22:51:21,413 13353 server.py[line:126] INFO: FastLLM API request thread pool: workers=32, prewarmed=32
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:18080 (Press CTRL+C to quit)
auth
INFO: 127.0.0.1:38806 - "GET /v1/models HTTP/1.1" 200 OK
auth
INFO: 127.0.0.1:38814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:51:22,413 13353 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-77440db9645b4a0b9a26ec756cd616ce
auth
INFO: 127.0.0.1:38820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:51:22,618 13353 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-e12bd13850b14db097a420ff1ab9ed6b
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [13353]
server_ready: true
warmup: ttft=0.205s, prompt_tokens=33, completion_tokens=1

benchmark_result
prompt_tokens: 62
completion_tokens: 1
finish_reason: length
ttft_seconds: 0.204260
total_seconds: 0.204372
decode_seconds: 0.000112
prefill_tokens_per_second: 303.53
decode_tokens_per_second: 8909.24
output_preview: I

{
"prompt_tokens": 62,
"completion_tokens": 1,
"total_tokens": 63,
"ttft": 0.2042601816356182,
"total_time": 0.20437242463231087,
"decode_time": 0.00011224299669265747,
"prefill_speed": 303.5344407487232,
"decode_speed": 8909.241818785264,
"finish_reason": "length",
"output_preview": "I",
"case_name": "w4a8-prefill-128",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 128,
"max_tokens": 1,
"prefill_length_chars": 128
}

[2/3] start_case: w4a8-prefill-1024
FastLLM Prefill benchmark
case_name: w4a8-prefill-1024
repo: /home/vllm_fastllm
base_url: http://127.0.0.1:18080
model_name: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
model_path: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
prefill_length_chars: 1024
prompt_chars: 1024
max_tokens: 1

2026-07-30 22:51:24,022 13457 server.py[line:305] INFO: Namespace(model=None, path='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', threads=16, low=False, dtype='bfloat16', moe_dtype='', moe_atype='', atype='bfloat16', kv_cache_dtype='auto', cuda_embedding=False, kv_cache_limit='auto', max_batch=1, chunked_prefill_size=-1, device='cuda', tp='', moe_device='numa', moe_device_layers=-1, moe_experts=-1, cache_history='', cache_fast='', enable_thinking='', cuda_shared_expert='true', enable_amx='false', tokens=-1, page_size=-1, prefix_cache='', prefix_cache_snapshot_interval_pages=-1, prefix_cache_snapshot_max_per_request=-1, prefix_cache_snapshot_max_records=-1, gpu_mem_ratio=0.9, cuda_slab=0, mtp=0, speculative_algorithm='', speculative_draft_model_path='', speculative_dspark_block_size=-1, speculative_dspark_confidence_threshold=0.5, triton=False, custom='', lora='', cache_dir='', dtype_config='', ori='', tool_call_parser='auto', chat_template='', model_name='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', host='127.0.0.1', port=18080, api_key='no-key', max_context_length=-1, temperature=None, top_p=None, top_k=None, repeat_penalty=None, think='false', hide_input=True, dev_mode=False, startup_progress='off', config='/tmp/w4a8_prefill_config.json', prompt_repeat=2048, prompt_unit='FastLLM prefill benchmark context block. ', question='请阅读以上上下文，并只回复“测试完成”。', max_tokens=1, startup_timeout=600, request_timeout=3600, warmup_max_tokens=1, skip_warmup=False, name='w4a8-prefill-1024', prefill_length=1024)
CPU Instruction Info: [AVX2: ON] [AVX512F: ON] [AVX512_VNNI: ON] [AVX512_BF16: ON] [AMX: ON]
[Fastllm] compressed-tensors W4A8: grouped 224 Linear weights.
Load 0
Load 0
Load 1
Load 1
Load 1
Load 2
Load 2
Load 2
Load 3
Load 3
Load 3
Load 4
Load 4
Load 4
Load 5
Load 5
Load 5
Load 6
Load 6
Load 6
Load 7
Load 7
Load 7
Load 8
Load 8
Load 8
Load 9
Load 9
Load 9
Load 10
Load 10
Load 10
Load 11
Load 11
Load 12
Load 12
Load 12
Load 13
Load 13
Load 13
Load 14
Load 14
Load 14
Load 15
Load 15
Load 15
Load 16
Load 16
Load 16
Load 17
Load 17
Load 17
Load 18
Load 18
Load 18
Load 19
Load 19
Load 19
Load 20
Load 20
Load 20
Load 21
Load 21
Load 21
Load 22
Load 22
Load 23
Load 23
Load 23
Load 24
Load 24
Load 24
Load 25
Load 25
Load 25
Load 26
Load 26
Load 26
Load 27
Load 27
Load 27
Load 28
Load 28
Load 28
Load 29
Load 29
Load 29
Load 30
Load 30
Load 30
Load 31
Load 31
Load 31
Load 32
Load 32
Load 32
Load 33
Load 33
Load 34
Load 34
Load 34
Load 35
Load 35
Load 35
Load 36
Load 36
Load 36
Load 37
Load 37
Load 37
Load 38
Load 38
Load 38
Load 39
Load 39
Load 39
Load 40
Load 40
Load 40
Load 41
Load 41
Load 41
Load 42
Load 42
Load 42
Load 43
Load 43
Load 43
Load 44
Load 44
Load 45
Load 45
Load 45
Load 46
Load 46
Load 46
Load 47
Load 47
Load 47
Load 48
Load 48
Load 48
Load 49
Load 49
Load 49
Load 50
Load 50
Load 50
Load 51
Load 51
Load 51
Load 52
Load 52
Load 52
Load 53
Load 53
Load 53
Load 54
Load 54
Load 54
Load 55
Load 55
Load 56
Load 56
Load 56
Load 57
Load 57
Load 57
Load 58
Load 58
Load 58
Load 59
Load 59
Load 59
Load 60
Load 60
Load 60
Load 61
Load 61
Load 61
Load 62
Load 62
Load 62
Load 63
Load 63
Load 63
Load 64
Load 64
Load 64
Load 65
Load 65
Load 65
Load 66
Load 66
Load 67
Load 67
Load 67
Load 68
Load 68
Load 68
Load 69
Load 69
Load 69
Load 70
Load 70
Load 70
Load 71
Load 71
Load 71
Load 72
Load 72
Load 72
Load 73
Load 73
Load 73
Load 74
Load 74
Load 74
Load 75
Load 75
Load 75
Load 76
Load 76
Load 76
Load 77
Load 77
Load 78
Load 78
Load 78
Load 79
Load 79
Load 79
Load 80
Load 80
Load 80
Load 81
Load 81
Load 81
Load 82
Load 82
Load 82
Load 83
Load 83
Load 83
Load 84
Load 84
Load 84
Load 85
Load 85
Load 85
Load 86
Load 86
Load 86
Load 87
Load 87
Load 87
Load 88
Load 88
Load 89
Load 89
Load 89
Load 90
Load 90
Load 90
Load 91
Load 91
Load 91
Load 92
Load 92
Load 92
Load 93
Load 93
Load 93
Load 94
Load 94
Load 94
Load 95
Load 95
Load 95
Load 96
Load 96
Load 96
Load 97
Load 97
Load 97
Load 98
Load 98
Load 98
Load 99
Load 99
Load 100
Loading 0
Loading 0
Loading 1
Loading 1
Loading 1
Loading 2
Loading 2
Loading 2
Loading 3
Loading 3
Loading 3
Loading 4
Loading 4
Loading 4
Loading 5
Loading 5
Loading 5
Loading 6
Loading 6
Loading 6
Loading 7
Loading 7
Loading 7
Loading 8
Loading 8
Loading 8
Loading 9
Loading 9
Loading 9
Loading 10
Loading 10
Loading 10
Loading 11
Loading 11
Loading 12
Loading 12
Loading 12
Loading 13
Loading 13
Loading 13
Loading 14
Loading 14
Loading 14
Loading 15
Loading 15
Loading 15
Loading 16
Loading 16
Loading 16
Loading 17
Loading 17
Loading 17
Loading 18
Loading 18
Loading 18
Loading 19
Loading 19
Loading 19
Loading 20
Loading 20
Loading 20
Loading 21
Loading 21
Loading 21
Loading 22
Loading 22
Loading 23
Loading 23
Loading 23
Loading 24
Loading 24
Loading 24
Loading 25
Loading 25
Loading 25
Loading 26
Loading 26
Loading 26
Loading 27
Loading 27
Loading 27
Loading 28
Loading 28
Loading 28
Loading 29
Loading 29
Loading 29
Loading 30
Loading 30
Loading 30
Loading 31
Loading 31
Loading 31
Loading 32
Loading 32
Loading 32
Loading 33
Loading 33
Loading 34
Loading 34
Loading 34
Loading 35
Loading 35
Loading 35
Loading 36
Loading 36
Loading 36
Loading 37
Loading 37
Loading 37
Loading 38
Loading 38
Loading 38
Loading 39
Loading 39
Loading 39
Loading 40
Loading 40
Loading 40
Loading 41
Loading 41
Loading 41
Loading 42
Loading 42
Loading 42
Loading 43
Loading 43
Loading 43
Loading 44
Loading 44
Loading 45
Loading 45
Loading 45
Loading 46
Loading 46
Loading 46
Loading 47
Loading 47
Loading 47
Loading 48
Loading 48
Loading 48
Loading 49
Loading 49
Loading 49
Loading 50
Loading 50
Loading 50
Loading 51
Loading 51
Loading 51
Loading 52
Loading 52
Loading 52
Loading 53
Loading 53
Loading 53
Loading 54
Loading 54
Loading 54
Loading 55
Loading 55
Loading 56
Loading 56
Loading 56
Loading 57
Loading 57
Loading 57
Loading 58
Loading 58
Loading 58
Loading 59
Loading 59
Loading 59
Loading 60
Loading 60
Loading 60
Loading 61
Loading 61
Loading 61
Loading 62
Loading 62
Loading 62
Loading 63
Loading 63
Loading 63
Loading 64
Loading 64
Loading 64
Loading 65
Loading 65
Loading 65
Loading 66
Loading 66
Loading 67
Loading 67
Loading 67
Loading 68
Loading 68
Loading 68
Loading 69
Loading 69
Loading 69
Loading 70
Loading 70
Loading 70
Loading 71
Loading 71
Loading 71
Loading 72
Loading 72
Loading 72
Loading 73
Loading 73
Loading 73
Loading 74
Loading 74
Loading 74
Loading 75
Loading 75
Loading 75
Loading 76
Loading 76
Loading 76
Loading 77
Loading 77
Loading 78
Loading 78
Loading 78
Loading 79
Loading 79
Loading 79
Loading 80
Loading 80
Loading 80
Loading 81
Loading 81
Loading 81
Loading 82
Loading 82
Loading 82
Loading 83
Loading 83
Loading 83
Loading 84
Loading 84
Loading 84
Loading 85
Loading 85
Loading 85
Loading 86
Loading 86
Loading 86
Loading 87
Loading 87
Loading 87
Loading 88
Loading 88
Loading 89
Loading 89
Loading 89
Loading 90
Loading 90
Loading 90
Loading 91
Loading 91
Loading 91
Loading 92
Loading 92
Loading 92
Loading 93
Loading 93
Loading 93
Loading 94
Loading 94
Loading 94
Loading 95
Loading 95
Loading 95
Loading 96
Loading 96
Loading 96
Loading 97
Loading 97
Loading 97
Loading 98
Loading 98
Loading 98
Loading 99
Loading 99
Loading 100
Load libfastllm_tools.so
Load AutoTokenizer failed. (you can try install transformers)
Try load fastllm tokenizer.
[Fastllm] default generation config: {'repetition_penalty': 1.0, 'top_p': 0.9, 'top_k': 1, 'temperature': 0.6}
2026-07-30 22:51:35,611 13457 server.py[line:321] INFO: Model context window: 8192 tokens per session (model=8192, shared KV cache=None, configured limit=None)
Increased ulimit from 1024 to 65535.
INFO: Started server process [13457]
2026-07-30 22:51:35,635 13457 server.py[line:126] INFO: FastLLM API request thread pool: workers=32, prewarmed=32
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:18080 (Press CTRL+C to quit)
auth
INFO: 127.0.0.1:59522 - "GET /v1/models HTTP/1.1" 200 OK
auth
INFO: 127.0.0.1:59530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:51:36,383 13457 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-ef2d8799590a4306b445fcaf7b8003fd
auth
INFO: 127.0.0.1:59538 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:51:36,604 13457 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-1b80d90e03ca4e13b7b0783289c73316
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [13457]
server_ready: true
warmup: ttft=0.204s, prompt_tokens=33, completion_tokens=1

benchmark_result
prompt_tokens: 289
completion_tokens: 1
finish_reason: length
ttft_seconds: 0.220564
total_seconds: 0.220706
decode_seconds: 0.000142
prefill_tokens_per_second: 1310.28
decode_tokens_per_second: 7056.29
output_preview: It

{
"prompt_tokens": 289,
"completion_tokens": 1,
"total_tokens": 290,
"ttft": 0.22056427597999573,
"total_time": 0.22070599347352982,
"decode_time": 0.00014171749353408813,
"prefill_speed": 1310.2756496533061,
"decode_speed": 7056.291887913359,
"finish_reason": "length",
"output_preview": "It",
"case_name": "w4a8-prefill-1024",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 1024,
"max_tokens": 1,
"prefill_length_chars": 1024
}

[3/3] start_case: w4a8-prefill-4096
FastLLM Prefill benchmark
case_name: w4a8-prefill-4096
repo: /home/vllm_fastllm
base_url: http://127.0.0.1:18080
model_name: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
model_path: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
prefill_length_chars: 4096
prompt_chars: 4096
max_tokens: 1

2026-07-30 22:51:37,941 13560 server.py[line:305] INFO: Namespace(model=None, path='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', threads=16, low=False, dtype='bfloat16', moe_dtype='', moe_atype='', atype='bfloat16', kv_cache_dtype='auto', cuda_embedding=False, kv_cache_limit='auto', max_batch=1, chunked_prefill_size=-1, device='cuda', tp='', moe_device='numa', moe_device_layers=-1, moe_experts=-1, cache_history='', cache_fast='', enable_thinking='', cuda_shared_expert='true', enable_amx='false', tokens=-1, page_size=-1, prefix_cache='', prefix_cache_snapshot_interval_pages=-1, prefix_cache_snapshot_max_per_request=-1, prefix_cache_snapshot_max_records=-1, gpu_mem_ratio=0.9, cuda_slab=0, mtp=0, speculative_algorithm='', speculative_draft_model_path='', speculative_dspark_block_size=-1, speculative_dspark_confidence_threshold=0.5, triton=False, custom='', lora='', cache_dir='', dtype_config='', ori='', tool_call_parser='auto', chat_template='', model_name='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', host='127.0.0.1', port=18080, api_key='no-key', max_context_length=-1, temperature=None, top_p=None, top_k=None, repeat_penalty=None, think='false', hide_input=True, dev_mode=False, startup_progress='off', config='/tmp/w4a8_prefill_config.json', prompt_repeat=2048, prompt_unit='FastLLM prefill benchmark context block. ', question='请阅读以上上下文，并只回复“测试完成”。', max_tokens=1, startup_timeout=600, request_timeout=3600, warmup_max_tokens=1, skip_warmup=False, name='w4a8-prefill-4096', prefill_length=4096)
CPU Instruction Info: [AVX2: ON] [AVX512F: ON] [AVX512_VNNI: ON] [AVX512_BF16: ON] [AMX: ON]
[Fastllm] compressed-tensors W4A8: grouped 224 Linear weights.
Load 0
Load 0
Load 1
Load 1
Load 1
Load 2
Load 2
Load 2
Load 3
Load 3
Load 3
Load 4
Load 4
Load 4
Load 5
Load 5
Load 5
Load 6
Load 6
Load 6
Load 7
Load 7
Load 7
Load 8
Load 8
Load 8
Load 9
Load 9
Load 9
Load 10
Load 10
Load 10
Load 11
Load 11
Load 12
Load 12
Load 12
Load 13
Load 13
Load 13
Load 14
Load 14
Load 14
Load 15
Load 15
Load 15
Load 16
Load 16
Load 16
Load 17
Load 17
Load 17
Load 18
Load 18
Load 18
Load 19
Load 19
Load 19
Load 20
Load 20
Load 20
Load 21
Load 21
Load 21
Load 22
Load 22
Load 23
Load 23
Load 23
Load 24
Load 24
Load 24
Load 25
Load 25
Load 25
Load 26
Load 26
Load 26
Load 27
Load 27
Load 27
Load 28
Load 28
Load 28
Load 29
Load 29
Load 29
Load 30
Load 30
Load 30
Load 31
Load 31
Load 31
Load 32
Load 32
Load 32
Load 33
Load 33
Load 34
Load 34
Load 34
Load 35
Load 35
Load 35
Load 36
Load 36
Load 36
Load 37
Load 37
Load 37
Load 38
Load 38
Load 38
Load 39
Load 39
Load 39
Load 40
Load 40
Load 40
Load 41
Load 41
Load 41
Load 42
Load 42
Load 42
Load 43
Load 43
Load 43
Load 44
Load 44
Load 45
Load 45
Load 45
Load 46
Load 46
Load 46
Load 47
Load 47
Load 47
Load 48
Load 48
Load 48
Load 49
Load 49
Load 49
Load 50
Load 50
Load 50
Load 51
Load 51
Load 51
Load 52
Load 52
Load 52
Load 53
Load 53
Load 53
Load 54
Load 54
Load 54
Load 55
Load 55
Load 56
Load 56
Load 56
Load 57
Load 57
Load 57
Load 58
Load 58
Load 58
Load 59
Load 59
Load 59
Load 60
Load 60
Load 60
Load 61
Load 61
Load 61
Load 62
Load 62
Load 62
Load 63
Load 63
Load 63
Load 64
Load 64
Load 64
Load 65
Load 65
Load 65
Load 66
Load 66
Load 67
Load 67
Load 67
Load 68
Load 68
Load 68
Load 69
Load 69
Load 69
Load 70
Load 70
Load 70
Load 71
Load 71
Load 71
Load 72
Load 72
Load 72
Load 73
Load 73
Load 73
Load 74
Load 74
Load 74
Load 75
Load 75
Load 75
Load 76
Load 76
Load 76
Load 77
Load 77
Load 78
Load 78
Load 78
Load 79
Load 79
Load 79
Load 80
Load 80
Load 80
Load 81
Load 81
Load 81
Load 82
Load 82
Load 82
Load 83
Load 83
Load 83
Load 84
Load 84
Load 84
Load 85
Load 85
Load 85
Load 86
Load 86
Load 86
Load 87
Load 87
Load 87
Load 88
Load 88
Load 89
Load 89
Load 89
Load 90
Load 90
Load 90
Load 91
Load 91
Load 91
Load 92
Load 92
Load 92
Load 93
Load 93
Load 93
Load 94
Load 94
Load 94
Load 95
Load 95
Load 95
Load 96
Load 96
Load 96
Load 97
Load 97
Load 97
Load 98
Load 98
Load 98
Load 99
Load 99
Load 100
Loading 0
Loading 0
Loading 1
Loading 1
Loading 1
Loading 2
Loading 2
Loading 2
Loading 3
Loading 3
Loading 3
Loading 4
Loading 4
Loading 4
Loading 5
Loading 5
Loading 5
Loading 6
Loading 6
Loading 6
Loading 7
Loading 7
Loading 7
Loading 8
Loading 8
Loading 8
Loading 9
Loading 9
Loading 9
Loading 10
Loading 10
Loading 10
Loading 11
Loading 11
Loading 12
Loading 12
Loading 12
Loading 13
Loading 13
Loading 13
Loading 14
Loading 14
Loading 14
Loading 15
Loading 15
Loading 15
Loading 16
Loading 16
Loading 16
Loading 17
Loading 17
Loading 17
Loading 18
Loading 18
Loading 18
Loading 19
Loading 19
Loading 19
Loading 20
Loading 20
Loading 20
Loading 21
Loading 21
Loading 21
Loading 22
Loading 22
Loading 23
Loading 23
Loading 23
Loading 24
Loading 24
Loading 24
Loading 25
Loading 25
Loading 25
Loading 26
Loading 26
Loading 26
Loading 27
Loading 27
Loading 27
Loading 28
Loading 28
Loading 28
Loading 29
Loading 29
Loading 29
Loading 30
Loading 30
Loading 30
Loading 31
Loading 31
Loading 31
Loading 32
Loading 32
Loading 32
Loading 33
Loading 33
Loading 34
Loading 34
Loading 34
Loading 35
Loading 35
Loading 35
Loading 36
Loading 36
Loading 36
Loading 37
Loading 37
Loading 37
Loading 38
Loading 38
Loading 38
Loading 39
Loading 39
Loading 39
Loading 40
Loading 40
Loading 40
Loading 41
Loading 41
Loading 41
Loading 42
Loading 42
Loading 42
Loading 43
Loading 43
Loading 43
Loading 44
Loading 44
Loading 45
Loading 45
Loading 45
Loading 46
Loading 46
Loading 46
Loading 47
Loading 47
Loading 47
Loading 48
Loading 48
Loading 48
Loading 49
Loading 49
Loading 49
Loading 50
Loading 50
Loading 50
Loading 51
Loading 51
Loading 51
Loading 52
Loading 52
Loading 52
Loading 53
Loading 53
Loading 53
Loading 54
Loading 54
Loading 54
Loading 55
Loading 55
Loading 56
Loading 56
Loading 56
Loading 57
Loading 57
Loading 57
Loading 58
Loading 58
Loading 58
Loading 59
Loading 59
Loading 59
Loading 60
Loading 60
Loading 60
Loading 61
Loading 61
Loading 61
Loading 62
Loading 62
Loading 62
Loading 63
Loading 63
Loading 63
Loading 64
Loading 64
Loading 64
Loading 65
Loading 65
Loading 65
Loading 66
Loading 66
Loading 67
Loading 67
Loading 67
Loading 68
Loading 68
Loading 68
Loading 69
Loading 69
Loading 69
Loading 70
Loading 70
Loading 70
Loading 71
Loading 71
Loading 71
Loading 72
Loading 72
Loading 72
Loading 73
Loading 73
Loading 73
Loading 74
Loading 74
Loading 74
Loading 75
Loading 75
Loading 75
Loading 76
Loading 76
Loading 76
Loading 77
Loading 77
Loading 78
Loading 78
Loading 78
Loading 79
Loading 79
Loading 79
Loading 80
Loading 80
Loading 80
Loading 81
Loading 81
Loading 81
Loading 82
Loading 82
Loading 82
Loading 83
Loading 83
Loading 83
Loading 84
Loading 84
Loading 84
Loading 85
Loading 85
Loading 85
Loading 86
Loading 86
Loading 86
Loading 87
Loading 87
Loading 87
Loading 88
Loading 88
Loading 89
Loading 89
Loading 89
Loading 90
Loading 90
Loading 90
Loading 91
Loading 91
Loading 91
Loading 92
Loading 92
Loading 92
Loading 93
Loading 93
Loading 93
Loading 94
Loading 94
Loading 94
Loading 95
Loading 95
Loading 95
Loading 96
Loading 96
Loading 96
Loading 97
Loading 97
Loading 97
Loading 98
Loading 98
Loading 98
Loading 99
Loading 99
Loading 100
Load libfastllm_tools.so
Load AutoTokenizer failed. (you can try install transformers)
Try load fastllm tokenizer.
[Fastllm] default generation config: {'repetition_penalty': 1.0, 'top_p': 0.9, 'top_k': 1, 'temperature': 0.6}
2026-07-30 22:51:49,137 13560 server.py[line:321] INFO: Model context window: 8192 tokens per session (model=8192, shared KV cache=None, configured limit=None)
Increased ulimit from 1024 to 65535.
INFO: Started server process [13560]
2026-07-30 22:51:49,160 13560 server.py[line:126] INFO: FastLLM API request thread pool: workers=32, prewarmed=32
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:18080 (Press CTRL+C to quit)
auth
INFO: 127.0.0.1:52914 - "GET /v1/models HTTP/1.1" 200 OK
auth
INFO: 127.0.0.1:52924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:51:50,295 13560 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-0e703a4dadbc484796fb1a93a2961411
auth
INFO: 127.0.0.1:52940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:51:50,556 13560 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-762ca1d37939411288f113a74255b150
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [13560]
server_ready: true
warmup: ttft=0.197s, prompt_tokens=33, completion_tokens=1

benchmark_result
prompt_tokens: 1084
completion_tokens: 1
finish_reason: length
ttft_seconds: 0.260873
total_seconds: 0.261014
decode_seconds: 0.000142
prefill_tokens_per_second: 4155.28
decode_tokens_per_second: 7062.60
output_preview: It

{
"prompt_tokens": 1084,
"completion_tokens": 1,
"total_tokens": 1085,
"ttft": 0.26087280362844467,
"total_time": 0.2610143944621086,
"decode_time": 0.00014159083366394043,
"prefill_speed": 4155.281750043661,
"decode_speed": 7062.604083350873,
"finish_reason": "length",
"output_preview": "It",
"case_name": "w4a8-prefill-4096",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 4096,
"max_tokens": 1,
"prefill_length_chars": 4096
}

============================================================
benchmark_summary
[
{
"prompt_tokens": 62,
"completion_tokens": 1,
"total_tokens": 63,
"ttft": 0.2042601816356182,
"total_time": 0.20437242463231087,
"decode_time": 0.00011224299669265747,
"prefill_speed": 303.5344407487232,
"decode_speed": 8909.241818785264,
"finish_reason": "length",
"output_preview": "I",
"case_name": "w4a8-prefill-128",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 128,
"max_tokens": 1,
"prefill_length_chars": 128
},
{
"prompt_tokens": 289,
"completion_tokens": 1,
"total_tokens": 290,
"ttft": 0.22056427597999573,
"total_time": 0.22070599347352982,
"decode_time": 0.00014171749353408813,
"prefill_speed": 1310.2756496533061,
"decode_speed": 7056.291887913359,
"finish_reason": "length",
"output_preview": "It",
"case_name": "w4a8-prefill-1024",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 1024,
"max_tokens": 1,
"prefill_length_chars": 1024
},
{
"prompt_tokens": 1084,
"completion_tokens": 1,
"total_tokens": 1085,
"ttft": 0.26087280362844467,
"total_time": 0.2610143944621086,
"decode_time": 0.00014159083366394043,
"prefill_speed": 4155.281750043661,
"decode_speed": 7062.604083350873,
"finish_reason": "length",
"output_preview": "It",
"case_name": "w4a8-prefill-4096",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 4096,
"max_tokens": 1,
"prefill_length_chars": 4096
}
]

w4a8 decode性能测试

[1/2] start_case: w4a8-decode-batch1
FastLLM Decode benchmark
case_name: w4a8-decode-batch1
repo: /home/vllm_fastllm
base_url: http://127.0.0.1:18080
model_name: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
model_path: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
batch_size: 1
server_max_batch: 1
request_stagger_ms: 0
prefill_length_chars: 128
prompt_chars: 128
max_tokens_per_request: 256

2026-07-30 22:51:52,010 13668 server.py[line:305] INFO: Namespace(model=None, path='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', threads=16, low=False, dtype='bfloat16', moe_dtype='', moe_atype='', atype='bfloat16', kv_cache_dtype='auto', cuda_embedding=False, kv_cache_limit='auto', max_batch=1, chunked_prefill_size=-1, device='cuda', tp='', moe_device='numa', moe_device_layers=-1, moe_experts=-1, cache_history='', cache_fast='', enable_thinking='', cuda_shared_expert='true', enable_amx='false', tokens=-1, page_size=-1, prefix_cache='', prefix_cache_snapshot_interval_pages=-1, prefix_cache_snapshot_max_per_request=-1, prefix_cache_snapshot_max_records=-1, gpu_mem_ratio=0.9, cuda_slab=0, mtp=0, speculative_algorithm='', speculative_draft_model_path='', speculative_dspark_block_size=-1, speculative_dspark_confidence_threshold=0.5, triton=False, custom='', lora='', cache_dir='', dtype_config='', ori='', tool_call_parser='auto', chat_template='', model_name='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', host='127.0.0.1', port=18080, api_key='no-key', max_context_length=-1, temperature=None, top_p=None, top_k=None, repeat_penalty=None, think='false', hide_input=True, dev_mode=False, startup_progress='off', config='/tmp/w4a8_decode_config.json', batch_size=1, prompt_repeat=16, prompt_unit='FastLLM decode benchmark context block. ', question='请连续输出数字序列，不要解释。', prefill_length=128, max_tokens=256, startup_timeout=600, request_timeout=3600, warmup_max_tokens=8, request_stagger_ms=0, skip_warmup=False, name='w4a8-decode-batch1')
CPU Instruction Info: [AVX2: ON] [AVX512F: ON] [AVX512_VNNI: ON] [AVX512_BF16: ON] [AMX: ON]
[Fastllm] compressed-tensors W4A8: grouped 224 Linear weights.
Load 0
Load 0
Load 1
Load 1
Load 1
Load 2
Load 2
Load 2
Load 3
Load 3
Load 3
Load 4
Load 4
Load 4
Load 5
Load 5
Load 5
Load 6
Load 6
Load 6
Load 7
Load 7
Load 7
Load 8
Load 8
Load 8
Load 9
Load 9
Load 9
Load 10
Load 10
Load 10
Load 11
Load 11
Load 12
Load 12
Load 12
Load 13
Load 13
Load 13
Load 14
Load 14
Load 14
Load 15
Load 15
Load 15
Load 16
Load 16
Load 16
Load 17
Load 17
Load 17
Load 18
Load 18
Load 18
Load 19
Load 19
Load 19
Load 20
Load 20
Load 20
Load 21
Load 21
Load 21
Load 22
Load 22
Load 23
Load 23
Load 23
Load 24
Load 24
Load 24
Load 25
Load 25
Load 25
Load 26
Load 26
Load 26
Load 27
Load 27
Load 27
Load 28
Load 28
Load 28
Load 29
Load 29
Load 29
Load 30
Load 30
Load 30
Load 31
Load 31
Load 31
Load 32
Load 32
Load 32
Load 33
Load 33
Load 34
Load 34
Load 34
Load 35
Load 35
Load 35
Load 36
Load 36
Load 36
Load 37
Load 37
Load 37
Load 38
Load 38
Load 38
Load 39
Load 39
Load 39
Load 40
Load 40
Load 40
Load 41
Load 41
Load 41
Load 42
Load 42
Load 42
Load 43
Load 43
Load 43
Load 44
Load 44
Load 45
Load 45
Load 45
Load 46
Load 46
Load 46
Load 47
Load 47
Load 47
Load 48
Load 48
Load 48
Load 49
Load 49
Load 49
Load 50
Load 50
Load 50
Load 51
Load 51
Load 51
Load 52
Load 52
Load 52
Load 53
Load 53
Load 53
Load 54
Load 54
Load 54
Load 55
Load 55
Load 56
Load 56
Load 56
Load 57
Load 57
Load 57
Load 58
Load 58
Load 58
Load 59
Load 59
Load 59
Load 60
Load 60
Load 60
Load 61
Load 61
Load 61
Load 62
Load 62
Load 62
Load 63
Load 63
Load 63
Load 64
Load 64
Load 64
Load 65
Load 65
Load 65
Load 66
Load 66
Load 67
Load 67
Load 67
Load 68
Load 68
Load 68
Load 69
Load 69
Load 69
Load 70
Load 70
Load 70
Load 71
Load 71
Load 71
Load 72
Load 72
Load 72
Load 73
Load 73
Load 73
Load 74
Load 74
Load 74
Load 75
Load 75
Load 75
Load 76
Load 76
Load 76
Load 77
Load 77
Load 78
Load 78
Load 78
Load 79
Load 79
Load 79
Load 80
Load 80
Load 80
Load 81
Load 81
Load 81
Load 82
Load 82
Load 82
Load 83
Load 83
Load 83
Load 84
Load 84
Load 84
Load 85
Load 85
Load 85
Load 86
Load 86
Load 86
Load 87
Load 87
Load 87
Load 88
Load 88
Load 89
Load 89
Load 89
Load 90
Load 90
Load 90
Load 91
Load 91
Load 91
Load 92
Load 92
Load 92
Load 93
Load 93
Load 93
Load 94
Load 94
Load 94
Load 95
Load 95
Load 95
Load 96
Load 96
Load 96
Load 97
Load 97
Load 97
Load 98
Load 98
Load 98
Load 99
Load 99
Load 100
Loading 0
Loading 0
Loading 1
Loading 1
Loading 1
Loading 2
Loading 2
Loading 2
Loading 3
Loading 3
Loading 3
Loading 4
Loading 4
Loading 4
Loading 5
Loading 5
Loading 5
Loading 6
Loading 6
Loading 6
Loading 7
Loading 7
Loading 7
Loading 8
Loading 8
Loading 8
Loading 9
Loading 9
Loading 9
Loading 10
Loading 10
Loading 10
Loading 11
Loading 11
Loading 12
Loading 12
Loading 12
Loading 13
Loading 13
Loading 13
Loading 14
Loading 14
Loading 14
Loading 15
Loading 15
Loading 15
Loading 16
Loading 16
Loading 16
Loading 17
Loading 17
Loading 17
Loading 18
Loading 18
Loading 18
Loading 19
Loading 19
Loading 19
Loading 20
Loading 20
Loading 20
Loading 21
Loading 21
Loading 21
Loading 22
Loading 22
Loading 23
Loading 23
Loading 23
Loading 24
Loading 24
Loading 24
Loading 25
Loading 25
Loading 25
Loading 26
Loading 26
Loading 26
Loading 27
Loading 27
Loading 27
Loading 28
Loading 28
Loading 28
Loading 29
Loading 29
Loading 29
Loading 30
Loading 30
Loading 30
Loading 31
Loading 31
Loading 31
Loading 32
Loading 32
Loading 32
Loading 33
Loading 33
Loading 34
Loading 34
Loading 34
Loading 35
Loading 35
Loading 35
Loading 36
Loading 36
Loading 36
Loading 37
Loading 37
Loading 37
Loading 38
Loading 38
Loading 38
Loading 39
Loading 39
Loading 39
Loading 40
Loading 40
Loading 40
Loading 41
Loading 41
Loading 41
Loading 42
Loading 42
Loading 42
Loading 43
Loading 43
Loading 43
Loading 44
Loading 44
Loading 45
Loading 45
Loading 45
Loading 46
Loading 46
Loading 46
Loading 47
Loading 47
Loading 47
Loading 48
Loading 48
Loading 48
Loading 49
Loading 49
Loading 49
Loading 50
Loading 50
Loading 50
Loading 51
Loading 51
Loading 51
Loading 52
Loading 52
Loading 52
Loading 53
Loading 53
Loading 53
Loading 54
Loading 54
Loading 54
Loading 55
Loading 55
Loading 56
Loading 56
Loading 56
Loading 57
Loading 57
Loading 57
Loading 58
Loading 58
Loading 58
Loading 59
Loading 59
Loading 59
Loading 60
Loading 60
Loading 60
Loading 61
Loading 61
Loading 61
Loading 62
Loading 62
Loading 62
Loading 63
Loading 63
Loading 63
Loading 64
Loading 64
Loading 64
Loading 65
Loading 65
Loading 65
Loading 66
Loading 66
Loading 67
Loading 67
Loading 67
Loading 68
Loading 68
Loading 68
Loading 69
Loading 69
Loading 69
Loading 70
Loading 70
Loading 70
Loading 71
Loading 71
Loading 71
Loading 72
Loading 72
Loading 72
Loading 73
Loading 73
Loading 73
Loading 74
Loading 74
Loading 74
Loading 75
Loading 75
Loading 75
Loading 76
Loading 76
Loading 76
Loading 77
Loading 77
Loading 78
Loading 78
Loading 78
Loading 79
Loading 79
Loading 79
Loading 80
Loading 80
Loading 80
Loading 81
Loading 81
Loading 81
Loading 82
Loading 82
Loading 82
Loading 83
Loading 83
Loading 83
Loading 84
Loading 84
Loading 84
Loading 85
Loading 85
Loading 85
Loading 86
Loading 86
Loading 86
Loading 87
Loading 87
Loading 87
Loading 88
Loading 88
Loading 89
Loading 89
Loading 89
Loading 90
Loading 90
Loading 90
Loading 91
Loading 91
Loading 91
Loading 92
Loading 92
Loading 92
Loading 93
Loading 93
Loading 93
Loading 94
Loading 94
Loading 94
Loading 95
Loading 95
Loading 95
Loading 96
Loading 96
Loading 96
Loading 97
Loading 97
Loading 97
Loading 98
Loading 98
Loading 98
Loading 99
Loading 99
Loading 100
Load libfastllm_tools.so
Load AutoTokenizer failed. (you can try install transformers)
Try load fastllm tokenizer.
[Fastllm] default generation config: {'repetition_penalty': 1.0, 'top_p': 0.9, 'top_k': 1, 'temperature': 0.6}
2026-07-30 22:52:03,226 13668 server.py[line:321] INFO: Model context window: 8192 tokens per session (model=8192, shared KV cache=None, configured limit=None)
Increased ulimit from 1024 to 65535.
INFO: Started server process [13668]
2026-07-30 22:52:03,249 13668 server.py[line:126] INFO: FastLLM API request thread pool: workers=32, prewarmed=32
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:18080 (Press CTRL+C to quit)
auth
INFO: 127.0.0.1:60710 - "GET /v1/models HTTP/1.1" 200 OK
auth
INFO: 127.0.0.1:60720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:52:04,577 13668 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-8bab56dd6ac6449b99d15abbae969f52
auth
INFO: 127.0.0.1:60724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:52:07,511 13668 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-3e70028332344f129533e557e55135bf
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [13668]
server_ready: true
warmup: ttft=0.205s, completion_tokens=1, decode_speed=5.17 tok/s

benchmark_result
total_prompt_tokens: 62
total_completion_tokens: 14
batch_wall_seconds: 2.933819
batch_decode_seconds: 2.734685
launch_span_seconds: 0.000000
batch_decode_tokens_per_second: 5.12
end_to_end_tokens_per_second: 4.77
ttft_min_seconds: 0.199134
ttft_avg_seconds: 0.199134
ttft_max_seconds: 0.199134
per_request_decode_tokens_per_second: avg=5.12, min=5.12, max=5.12
finish_reason_counts: {"stop": 1}
sample_output_preview: [context-000002] FastLLM decode benchmark context block.

{
"requests": 1,
"request_stagger_ms": 0,
"total_prompt_tokens": 62,
"total_completion_tokens": 14,
"total_tokens": 76,
"batch_wall_time": 2.9338191226124763,
"batch_decode_time": 2.7346853390336037,
"batch_decode_speed": 5.119418969404132,
"end_to_end_speed": 4.771936992330129,
"ttft_min": 0.19913378357887268,
"ttft_avg": 0.19913378357887268,
"ttft_max": 0.19913378357887268,
"per_request_completion_tokens_avg": 14,
"per_request_total_time_avg": 2.9338191226124763,
"per_request_decode_time_avg": 2.7346853390336037,
"per_request_decode_speed_min": 5.119418969404132,
"per_request_decode_speed_avg": 5.119418969404132,
"per_request_decode_speed_max": 5.119418969404132,
"launch_span": 0.0,
"finish_reason_counts": {
"stop": 1
},
"request_results": [
{
"request_id": 0,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.19913378357887268,
"total_time": 2.9338191226124763,
"decode_time": 2.7346853390336037,
"decode_speed": 5.119418969404132,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
}
],
"case_name": "w4a8-decode-batch1",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 128,
"batch_size": 1,
"server_max_batch": 1,
"max_tokens": 256,
"queueing_expected": false,
"prefill_length_chars": 128
}

[2/2] start_case: w4a8-decode-batch4
FastLLM Decode benchmark
case_name: w4a8-decode-batch4
repo: /home/vllm_fastllm
base_url: http://127.0.0.1:18080
model_name: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
model_path: /root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ
batch_size: 4
server_max_batch: 4
request_stagger_ms: 0
prefill_length_chars: 128
prompt_chars: 128
max_tokens_per_request: 256

2026-07-30 22:52:08,860 13772 server.py[line:305] INFO: Namespace(model=None, path='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', threads=16, low=False, dtype='bfloat16', moe_dtype='', moe_atype='', atype='bfloat16', kv_cache_dtype='auto', cuda_embedding=False, kv_cache_limit='auto', max_batch=4, chunked_prefill_size=-1, device='cuda', tp='', moe_device='numa', moe_device_layers=-1, moe_experts=-1, cache_history='', cache_fast='', enable_thinking='', cuda_shared_expert='true', enable_amx='false', tokens=-1, page_size=-1, prefix_cache='', prefix_cache_snapshot_interval_pages=-1, prefix_cache_snapshot_max_per_request=-1, prefix_cache_snapshot_max_records=-1, gpu_mem_ratio=0.9, cuda_slab=0, mtp=0, speculative_algorithm='', speculative_draft_model_path='', speculative_dspark_block_size=-1, speculative_dspark_confidence_threshold=0.5, triton=False, custom='', lora='', cache_dir='', dtype_config='', ori='', tool_call_parser='auto', chat_template='', model_name='/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ', host='127.0.0.1', port=18080, api_key='no-key', max_context_length=-1, temperature=None, top_p=None, top_k=None, repeat_penalty=None, think='false', hide_input=True, dev_mode=False, startup_progress='off', config='/tmp/w4a8_decode_config.json', batch_size=4, prompt_repeat=16, prompt_unit='FastLLM decode benchmark context block. ', question='请连续输出数字序列，不要解释。', prefill_length=128, max_tokens=256, startup_timeout=600, request_timeout=3600, warmup_max_tokens=8, request_stagger_ms=0, skip_warmup=False, name='w4a8-decode-batch4')
CPU Instruction Info: [AVX2: ON] [AVX512F: ON] [AVX512_VNNI: ON] [AVX512_BF16: ON] [AMX: ON]
[Fastllm] compressed-tensors W4A8: grouped 224 Linear weights.
Load 0
Load 0
Load 1
Load 1
Load 1
Load 2
Load 2
Load 2
Load 3
Load 3
Load 3
Load 4
Load 4
Load 4
Load 5
Load 5
Load 5
Load 6
Load 6
Load 6
Load 7
Load 7
Load 7
Load 8
Load 8
Load 8
Load 9
Load 9
Load 9
Load 10
Load 10
Load 10
Load 11
Load 11
Load 12
Load 12
Load 12
Load 13
Load 13
Load 13
Load 14
Load 14
Load 14
Load 15
Load 15
Load 15
Load 16
Load 16
Load 16
Load 17
Load 17
Load 17
Load 18
Load 18
Load 18
Load 19
Load 19
Load 19
Load 20
Load 20
Load 20
Load 21
Load 21
Load 21
Load 22
Load 22
Load 23
Load 23
Load 23
Load 24
Load 24
Load 24
Load 25
Load 25
Load 25
Load 26
Load 26
Load 26
Load 27
Load 27
Load 27
Load 28
Load 28
Load 28
Load 29
Load 29
Load 29
Load 30
Load 30
Load 30
Load 31
Load 31
Load 31
Load 32
Load 32
Load 32
Load 33
Load 33
Load 34
Load 34
Load 34
Load 35
Load 35
Load 35
Load 36
Load 36
Load 36
Load 37
Load 37
Load 37
Load 38
Load 38
Load 38
Load 39
Load 39
Load 39
Load 40
Load 40
Load 40
Load 41
Load 41
Load 41
Load 42
Load 42
Load 42
Load 43
Load 43
Load 43
Load 44
Load 44
Load 45
Load 45
Load 45
Load 46
Load 46
Load 46
Load 47
Load 47
Load 47
Load 48
Load 48
Load 48
Load 49
Load 49
Load 49
Load 50
Load 50
Load 50
Load 51
Load 51
Load 51
Load 52
Load 52
Load 52
Load 53
Load 53
Load 53
Load 54
Load 54
Load 54
Load 55
Load 55
Load 56
Load 56
Load 56
Load 57
Load 57
Load 57
Load 58
Load 58
Load 58
Load 59
Load 59
Load 59
Load 60
Load 60
Load 60
Load 61
Load 61
Load 61
Load 62
Load 62
Load 62
Load 63
Load 63
Load 63
Load 64
Load 64
Load 64
Load 65
Load 65
Load 65
Load 66
Load 66
Load 67
Load 67
Load 67
Load 68
Load 68
Load 68
Load 69
Load 69
Load 69
Load 70
Load 70
Load 70
Load 71
Load 71
Load 71
Load 72
Load 72
Load 72
Load 73
Load 73
Load 73
Load 74
Load 74
Load 74
Load 75
Load 75
Load 75
Load 76
Load 76
Load 76
Load 77
Load 77
Load 78
Load 78
Load 78
Load 79
Load 79
Load 79
Load 80
Load 80
Load 80
Load 81
Load 81
Load 81
Load 82
Load 82
Load 82
Load 83
Load 83
Load 83
Load 84
Load 84
Load 84
Load 85
Load 85
Load 85
Load 86
Load 86
Load 86
Load 87
Load 87
Load 87
Load 88
Load 88
Load 89
Load 89
Load 89
Load 90
Load 90
Load 90
Load 91
Load 91
Load 91
Load 92
Load 92
Load 92
Load 93
Load 93
Load 93
Load 94
Load 94
Load 94
Load 95
Load 95
Load 95
Load 96
Load 96
Load 96
Load 97
Load 97
Load 97
Load 98
Load 98
Load 98
Load 99
Load 99
Load 100
Loading 0
Loading 0
Loading 1
Loading 1
Loading 1
Loading 2
Loading 2
Loading 2
Loading 3
Loading 3
Loading 3
Loading 4
Loading 4
Loading 4
Loading 5
Loading 5
Loading 5
Loading 6
Loading 6
Loading 6
Loading 7
Loading 7
Loading 7
Loading 8
Loading 8
Loading 8
Loading 9
Loading 9
Loading 9
Loading 10
Loading 10
Loading 10
Loading 11
Loading 11
Loading 12
Loading 12
Loading 12
Loading 13
Loading 13
Loading 13
Loading 14
Loading 14
Loading 14
Loading 15
Loading 15
Loading 15
Loading 16
Loading 16
Loading 16
Loading 17
Loading 17
Loading 17
Loading 18
Loading 18
Loading 18
Loading 19
Loading 19
Loading 19
Loading 20
Loading 20
Loading 20
Loading 21
Loading 21
Loading 21
Loading 22
Loading 22
Loading 23
Loading 23
Loading 23
Loading 24
Loading 24
Loading 24
Loading 25
Loading 25
Loading 25
Loading 26
Loading 26
Loading 26
Loading 27
Loading 27
Loading 27
Loading 28
Loading 28
Loading 28
Loading 29
Loading 29
Loading 29
Loading 30
Loading 30
Loading 30
Loading 31
Loading 31
Loading 31
Loading 32
Loading 32
Loading 32
Loading 33
Loading 33
Loading 34
Loading 34
Loading 34
Loading 35
Loading 35
Loading 35
Loading 36
Loading 36
Loading 36
Loading 37
Loading 37
Loading 37
Loading 38
Loading 38
Loading 38
Loading 39
Loading 39
Loading 39
Loading 40
Loading 40
Loading 40
Loading 41
Loading 41
Loading 41
Loading 42
Loading 42
Loading 42
Loading 43
Loading 43
Loading 43
Loading 44
Loading 44
Loading 45
Loading 45
Loading 45
Loading 46
Loading 46
Loading 46
Loading 47
Loading 47
Loading 47
Loading 48
Loading 48
Loading 48
Loading 49
Loading 49
Loading 49
Loading 50
Loading 50
Loading 50
Loading 51
Loading 51
Loading 51
Loading 52
Loading 52
Loading 52
Loading 53
Loading 53
Loading 53
Loading 54
Loading 54
Loading 54
Loading 55
Loading 55
Loading 56
Loading 56
Loading 56
Loading 57
Loading 57
Loading 57
Loading 58
Loading 58
Loading 58
Loading 59
Loading 59
Loading 59
Loading 60
Loading 60
Loading 60
Loading 61
Loading 61
Loading 61
Loading 62
Loading 62
Loading 62
Loading 63
Loading 63
Loading 63
Loading 64
Loading 64
Loading 64
Loading 65
Loading 65
Loading 65
Loading 66
Loading 66
Loading 67
Loading 67
Loading 67
Loading 68
Loading 68
Loading 68
Loading 69
Loading 69
Loading 69
Loading 70
Loading 70
Loading 70
Loading 71
Loading 71
Loading 71
Loading 72
Loading 72
Loading 72
Loading 73
Loading 73
Loading 73
Loading 74
Loading 74
Loading 74
Loading 75
Loading 75
Loading 75
Loading 76
Loading 76
Loading 76
Loading 77
Loading 77
Loading 78
Loading 78
Loading 78
Loading 79
Loading 79
Loading 79
Loading 80
Loading 80
Loading 80
Loading 81
Loading 81
Loading 81
Loading 82
Loading 82
Loading 82
Loading 83
Loading 83
Loading 83
Loading 84
Loading 84
Loading 84
Loading 85
Loading 85
Loading 85
Loading 86
Loading 86
Loading 86
Loading 87
Loading 87
Loading 87
Loading 88
Loading 88
Loading 89
Loading 89
Loading 89
Loading 90
Loading 90
Loading 90
Loading 91
Loading 91
Loading 91
Loading 92
Loading 92
Loading 92
Loading 93
Loading 93
Loading 93
Loading 94
Loading 94
Loading 94
Loading 95
Loading 95
Loading 95
Loading 96
Loading 96
Loading 96
Loading 97
Loading 97
Loading 97
Loading 98
Loading 98
Loading 98
Loading 99
Loading 99
Loading 100
Load libfastllm_tools.so
Load AutoTokenizer failed. (you can try install transformers)
Try load fastllm tokenizer.
[Fastllm] default generation config: {'repetition_penalty': 1.0, 'top_p': 0.9, 'top_k': 1, 'temperature': 0.6}
2026-07-30 22:52:20,060 13772 server.py[line:321] INFO: Model context window: 8192 tokens per session (model=8192, shared KV cache=None, configured limit=None)
Increased ulimit from 1024 to 65535.
INFO: Started server process [13772]
2026-07-30 22:52:20,082 13772 server.py[line:126] INFO: FastLLM API request thread pool: workers=32, prewarmed=32
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:18080 (Press CTRL+C to quit)
auth
INFO: 127.0.0.1:55628 - "GET /v1/models HTTP/1.1" 200 OK
auth
INFO: 127.0.0.1:55644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:52:21,425 13772 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-313ca7f5f06b4945ae113381d8d5e314
auth
auth
auth
auth
INFO: 127.0.0.1:55654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55656 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
INFO: 127.0.0.1:55658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2026-07-30 22:52:24,982 13772 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-cea916860c224059a0fc2c67ecb63819
2026-07-30 22:52:24,982 13772 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-3b29d0eac54546129ad71326b7112b40
2026-07-30 22:52:24,982 13772 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-c139300f76fb458286c4786fcba63768
2026-07-30 22:52:24,982 13772 fastllm_completion.py[line:2554] INFO: Abort request: fastllm-/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ-e89f76287e004f4c8bf42f127606755c
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [13772]
server_ready: true
warmup: ttft=0.207s, completion_tokens=1, decode_speed=5.22 tok/s

benchmark_result
total_prompt_tokens: 248
total_completion_tokens: 56
batch_wall_seconds: 3.556783
batch_decode_seconds: 3.355246
launch_span_seconds: 0.002473
batch_decode_tokens_per_second: 16.69
end_to_end_tokens_per_second: 15.74
ttft_min_seconds: 0.201536
ttft_avg_seconds: 0.494534
ttft_max_seconds: 0.787833
per_request_decode_tokens_per_second: avg=4.60, min=4.17, max=5.06
finish_reason_counts: {"stop": 4}
sample_output_preview: [context-000002] FastLLM decode benchmark context block.

{
"requests": 4,
"request_stagger_ms": 0,
"total_prompt_tokens": 248,
"total_completion_tokens": 56,
"total_tokens": 304,
"batch_wall_time": 3.5567827485501766,
"batch_decode_time": 3.3552464582026005,
"batch_decode_speed": 16.690279148673657,
"end_to_end_speed": 15.744565794137086,
"ttft_min": 0.20153629034757614,
"ttft_avg": 0.4945338014513254,
"ttft_max": 0.7878325097262859,
"per_request_completion_tokens_avg": 14,
"per_request_total_time_avg": 3.5554768973961473,
"per_request_decode_time_avg": 3.060943095944822,
"per_request_decode_speed_min": 4.17267976703063,
"per_request_decode_speed_avg": 4.59736311415228,
"per_request_decode_speed_max": 5.0590124509576455,
"launch_span": 0.0024726502597332,
"finish_reason_counts": {
"stop": 4
},
"request_results": [
{
"request_id": 0,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.20153629034757614,
"total_time": 3.5566943138837814,
"decode_time": 3.3551580235362053,
"decode_speed": 4.17267976703063,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
},
{
"request_id": 1,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.3975272439420223,
"total_time": 3.5559125915169716,
"decode_time": 3.1583853475749493,
"decode_speed": 4.43264467736636,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
},
{
"request_id": 2,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.7878325097262859,
"total_time": 3.555171024054289,
"decode_time": 2.767338514328003,
"decode_speed": 5.0590124509576455,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
},
{
"request_id": 3,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.5912391617894173,
"total_time": 3.554129660129547,
"decode_time": 2.96289049834013,
"decode_speed": 4.7251155612544835,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
}
],
"case_name": "w4a8-decode-batch4",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 128,
"batch_size": 4,
"server_max_batch": 4,
"max_tokens": 256,
"queueing_expected": false,
"prefill_length_chars": 128
}

============================================================
benchmark_summary
[
{
"requests": 1,
"request_stagger_ms": 0,
"total_prompt_tokens": 62,
"total_completion_tokens": 14,
"total_tokens": 76,
"batch_wall_time": 2.9338191226124763,
"batch_decode_time": 2.7346853390336037,
"batch_decode_speed": 5.119418969404132,
"end_to_end_speed": 4.771936992330129,
"ttft_min": 0.19913378357887268,
"ttft_avg": 0.19913378357887268,
"ttft_max": 0.19913378357887268,
"per_request_completion_tokens_avg": 14,
"per_request_total_time_avg": 2.9338191226124763,
"per_request_decode_time_avg": 2.7346853390336037,
"per_request_decode_speed_min": 5.119418969404132,
"per_request_decode_speed_avg": 5.119418969404132,
"per_request_decode_speed_max": 5.119418969404132,
"launch_span": 0.0,
"finish_reason_counts": {
"stop": 1
},
"request_results": [
{
"request_id": 0,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.19913378357887268,
"total_time": 2.9338191226124763,
"decode_time": 2.7346853390336037,
"decode_speed": 5.119418969404132,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
}
],
"case_name": "w4a8-decode-batch1",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 128,
"batch_size": 1,
"server_max_batch": 1,
"max_tokens": 256,
"queueing_expected": false,
"prefill_length_chars": 128
},
{
"requests": 4,
"request_stagger_ms": 0,
"total_prompt_tokens": 248,
"total_completion_tokens": 56,
"total_tokens": 304,
"batch_wall_time": 3.5567827485501766,
"batch_decode_time": 3.3552464582026005,
"batch_decode_speed": 16.690279148673657,
"end_to_end_speed": 15.744565794137086,
"ttft_min": 0.20153629034757614,
"ttft_avg": 0.4945338014513254,
"ttft_max": 0.7878325097262859,
"per_request_completion_tokens_avg": 14,
"per_request_total_time_avg": 3.5554768973961473,
"per_request_decode_time_avg": 3.060943095944822,
"per_request_decode_speed_min": 4.17267976703063,
"per_request_decode_speed_avg": 4.59736311415228,
"per_request_decode_speed_max": 5.0590124509576455,
"launch_span": 0.0024726502597332,
"finish_reason_counts": {
"stop": 4
},
"request_results": [
{
"request_id": 0,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.20153629034757614,
"total_time": 3.5566943138837814,
"decode_time": 3.3551580235362053,
"decode_speed": 4.17267976703063,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
},
{
"request_id": 1,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.3975272439420223,
"total_time": 3.5559125915169716,
"decode_time": 3.1583853475749493,
"decode_speed": 4.43264467736636,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
},
{
"request_id": 2,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.7878325097262859,
"total_time": 3.555171024054289,
"decode_time": 2.767338514328003,
"decode_speed": 5.0590124509576455,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
},
{
"request_id": 3,
"prompt_tokens": 62,
"completion_tokens": 14,
"total_tokens": 76,
"ttft": 0.5912391617894173,
"total_time": 3.554129660129547,
"decode_time": 2.96289049834013,
"decode_speed": 4.7251155612544835,
"finish_reason": "stop",
"output_preview": "[context-000002] FastLLM decode benchmark context block."
}
],
"case_name": "w4a8-decode-batch4",
"model_name": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"model_path": "/root/autodl-tmp/Meta-Llama-3-8B-Instruct-W4AFP8-AWQ",
"prompt_chars": 128,
"batch_size": 4,
"server_max_batch": 4,
"max_tokens": 256,
"queueing_expected": false,
"prefill_length_chars": 128
}
]

@moguizhizi

moguizhizi commented Jul 30, 2026

Copy link
Copy Markdown
Author
fp量化算子功能 性能测试

CPU Instruction Info: [AVX2: ON] [AVX512F: ON] [AVX512_VNNI: ON] [AVX512_BF16: ON] [AMX: ON]

== op: w4a8_activation_quant ==
dynamic per-token FP8 E4M3 activation quantization for W4A8
tokens=16 # number of flattened tokens
hidden=4096 # hidden size per token
dtype=bfloat16 # input datatype: bfloat16 or float16
[cuda:0] PASS
accuracy: max_abs_diff=0.000000e+00, max_rel_diff=0.000000e+00
latency: avg_ms=0.0121
throughput: io_speed=16.2218 GB/s, compute_speed=n/a

Summary: PASS

@moguizhizi

moguizhizi commented Jul 30, 2026

Copy link
Copy Markdown
Author
w4a8 moe 算子功能 性能测试

CPU Instruction Info: [AVX2: ON] [AVX512F: ON] [AVX512_VNNI: ON] [AVX512_BF16: ON] [AMX: ON]

== op: mergemoe_w4a8 ==
CUTLASS grouped W4A8 MoE with routed SwiGLU experts
batch=32 # token batch size
hidden=1024 # model hidden size
inter=1024 # expert intermediate size
experts=8 # number of routed experts
topk=2 # experts selected per token
[cuda:0] PASS
accuracy: max_abs_diff=0.000000e+00, max_rel_diff=0.000000e+00
latency: avg_ms=1.9440
throughput: io_speed=6.7087 GB/s, compute_speed=207.1257 GOPS

Summary: PASS

@moguizhizi

moguizhizi commented Jul 30, 2026

Copy link
Copy Markdown
Author

w4a8 moe 整体性能测试
benchmark-w4a8-moe-decode-batch1.log
benchmark-w4a8-moe-decode-batch4.log
benchmark-w4a8-moe-prefill.log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@moguizhizi