FP8 and NVFP4 quantization

[fengwang]

I just noticed FP8 and NVFP4 checkpoints are not yet officially implemented.

A few weeks ago I quantized the nano model to FP8 and NVFP4, and found the inference quality is nearly identical to the BF16 checkpoint. I also implemented a WebUI for them, and share here in case some are interested.

FP8 and NVFP4 quantization:

• WebUI
  • reference doc
  • walkthrough
• FP8 Blockwise, [128, 128] block:
  • checkpoint
  • examples
• NVFP4:
  • checkpoint
  • examples

[lfengad]

Thank @fengwang for the contribution! And we also have internal efforts on the FP8 models which will be released soon. Your provided ones would also be a good reference. Thanks again for the contribution!