[Feature] Support float8 dtype storage and deepseek v3 with fp8 inference. #9906

ZHUI · 2025-02-19T12:15:10Z

Before submitting

Lint code. If there are lint issues, please format the code first.

# Install and register `pre-commit` in the project folder
pip install pre-commit && pre-commit install

# Process previous code files separately
pre-commit run --file XXXX.py

Add test cases into tests folder. If there are codecov issues, please add tests cases first.

PR types

New features

PR changes

Others

Description

Support float8 dtype storage.

FP8的模型有： deepseek-ai/DeepSeek-V3-FP8, deepseek-ai/DeepSeek-R1-FP8

For FP8,

import paddle
paddle.set_default_dtype("bfloat16")
from paddlenlp.transformers import AutoModelForCausalLM, AutoConfig

path = "deepseek-ai/DeepSeek-V3-FP8"
config = AutoConfig.from_pretrained(path)
config.num_hidden_layers = 4
config.dtype = paddle.bfloat16
config.use_fp8 = True
model =  AutoModelForCausalLM.from_pretrained(path, config=config)
model.eval()
ret = model(input_ids= paddle.to_tensor([[10,11,12,13,14,15]], dtype=paddle.int64), return_dict=True)
print(ret)
# Cast bfloat16 result
# CausalLMOutputWithPast(loss=None, logits=Tensor(shape=[1, 5, 129280], dtype=bfloat16, place=Place(gpu:0), stop_gradient=False,
#        [[[ 4.40625000, -2.35937500, -0.38476562, ..., -0.24511719,
#           -0.08007812, -0.36523438],
#          [ 6.78125000, -0.89453125,  0.35937500, ...,  0.34570312,
#            0.52343750,  0.54296875],
#          [ 7.46875000,  1.75000000,  0.18457031, ...,  0.07324219,
#            0.32812500,  0.16406250],
#          [ 5.06250000, -3.82812500,  0.49609375, ...,  0.35546875,
#            0.39257812,  0.31445312],
#          [-0.67578125, -2.10937500,  0.05322266, ...,  0.40625000,
#            0.24023438,  0.01519775]]]), past_key_values=None, hidden_states=None, attentions=None)

# Pure FP8 kernel result

For BFLOAT16

import paddle
paddle.set_default_dtype("bfloat16")
from paddlenlp.transformers import AutoModelForCausalLM, AutoConfig

path = "deepseek-ai/DeepSeek-V3"
config = AutoConfig.from_pretrained(path)
config.num_hidden_layers = 4
config.dtype = paddle.bfloat16
model =  AutoModelForCausalLM.from_pretrained(path, config=config)
model.eval()
ret = model(input_ids= paddle.to_tensor([[10,11,12,13,14,15]], dtype=paddle.int64), return_dict=True)
print(ret)
# CausalLMOutputWithPast(loss=None, logits=Tensor(shape=[1, 5, 129280], dtype=bfloat16, place=Place(gpu:0), stop_gradient=False,
#        [[[ 4.40625000, -2.35937500, -0.38476562, ..., -0.24511719,
#           -0.08007812, -0.36523438],
#          [ 6.78125000, -0.89453125,  0.35937500, ...,  0.34570312,
#            0.52343750,  0.54296875],
#          [ 7.46875000,  1.75000000,  0.18457031, ...,  0.07324219,
#            0.32812500,  0.16406250],
#          [ 5.06250000, -3.82812500,  0.49609375, ...,  0.35546875,
#            0.39257812,  0.31445312],
#          [-0.67578125, -2.10937500,  0.05322266, ...,  0.40625000,
#            0.24023438,  0.01519775]]]), past_key_values=None, hidden_states=None, attentions=None)

paddle-bot · 2025-02-19T12:15:17Z

Thanks for your contribution!

CLAassistant · 2025-02-19T12:15:17Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

tests/transformers/test_safetensors.py

codecov · 2025-02-19T12:51:19Z

Codecov Report

Attention: Patch coverage is 42.77108% with 190 lines in your changes missing coverage. Please review.

Project coverage is 51.08%. Comparing base (40c3530) to head (92b0a94).

❗ Current head 92b0a94 differs from pull request most recent head be36ba2

Please upload reports for the commit be36ba2 to get more accurate results.

Files with missing lines	Patch %	Lines
paddlenlp/transformers/deepseek_v2/kernel.py	15.47%	71 Missing ⚠️
paddlenlp/transformers/deepseek_v2/modeling.py	11.76%	45 Missing ⚠️
paddlenlp/transformers/deepseek_v2/fp8_linear.py	47.54%	32 Missing ⚠️
paddlenlp/mergekit/merge_model.py	15.00%	17 Missing ⚠️
paddlenlp/utils/paddle_patch.py	81.25%	15 Missing ⚠️
paddlenlp/transformers/model_utils.py	65.00%	7 Missing ⚠️
paddlenlp/transformers/moe_gate.py	60.00%	2 Missing ⚠️
...addlenlp/transformers/deepseek_v2/configuration.py	0.00%	1 Missing ⚠️

❌ Your patch check has failed because the patch coverage (42.77%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.
❌ Your project check has failed because the head coverage (51.08%) is below the target coverage (58.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #9906      +/-   ##
===========================================
- Coverage    51.08%   51.08%   -0.01%     
===========================================
  Files          745      748       +3     
  Lines       119274   119522     +248     
===========================================
+ Hits         60927    61053     +126     
- Misses       58347    58469     +122

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

DesmonDay · 2025-02-21T07:23:55Z

tests/transformers/test_safetensors.py

+    return paddle.to_tensor(tensor)
+
+
+class EextendDtypeNumpySafe(unittest.TestCase):


wawltor · 2025-02-27T07:01:54Z

paddlenlp/mergekit/merge_model.py

+                    is_bf16 = str(tensor.dtype) in ["uint16", "bfloat16"]
+                    tensor = paddle.Tensor.__call__(tensor, zero_copy=True)
+                    lora_A_tensor = paddle.Tensor.__call__(lora_A_tensor, zero_copy=True)
+                    lora_B_tensor = paddle.Tensor.__call__(lora_B_tensor, zero_copy=True)
                    if self.is_cpu and is_bf16:


这里替换__call__函数的原因是什么？

wawltor · 2025-02-27T07:09:41Z

paddlenlp/transformers/deepseek_v2/kernel.py

@@ -0,0 +1,226 @@
+# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.


这里的CopyRight是不是要加上deepseek

可以改成 kernel.py -> fp8_kernel.py

wawltor · 2025-02-27T07:21:57Z

paddlenlp/transformers/deepseek_v2/modeling.py

 from .configuration import DeepseekV2Config
+from .fp8_linear import Linear


这里可以直接import吗？看起来是把Linear全替换了

wawltor · 2025-02-27T07:24:42Z

paddlenlp/transformers/deepseek_v2/modeling.py

@@ -628,36 +635,43 @@ def __init__(self, config: DeepseekV2Config, hidden_size=None, intermediate_size
        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size

+        def linear_dtype_gaurd():


fp8参数的加载已经在在from_pretrained接口适配了？

是的，直接初始化成加载FP8参数。

suppport float8 dtype.

3189083

ZHUI commented Feb 19, 2025

View reviewed changes

tests/transformers/test_safetensors.py Outdated Show resolved Hide resolved

fix

c17aa5f

fix for empty shape.

f97937b

DesmonDay reviewed Feb 21, 2025

View reviewed changes

ZHUI added 14 commits February 24, 2025 20:44

Support load fp8 model.

b9dbed9

debug

60dab5f

[DeepSeek] Support for float8 loading and inference.

0f94c50

Add paddle patch.

10bb3a6

refine safetensors.

72f3013

refine code.

91f43cc

Merge remote-tracking branch 'upstream/develop' into support_fp8_storage

c536cc3

Fix moe dtype error.

e248bf8

fix paddle.Tensor.

bda020c

fix kwargs with patch api.

c16b2f3

fix for fast safe open

eb5f238

fix merge kit.

93b3c41

fix inference.

534f134

remove set device.

92b0a94

ZHUI changed the title ~~[Feature] Support float8 dtype storage.~~ [Feature] Support float8 dtype storage and deepseek v3 with fp18 inference. Feb 27, 2025

ZHUI changed the title ~~[Feature] Support float8 dtype storage and deepseek v3 with fp18 inference.~~ [Feature] Support float8 dtype storage and deepseek v3 with fp8 inference. Feb 27, 2025

wawltor reviewed Feb 27, 2025

View reviewed changes

Merge branch 'develop' into support_fp8_storage

be36ba2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support float8 dtype storage and deepseek v3 with fp8 inference. #9906

[Feature] Support float8 dtype storage and deepseek v3 with fp8 inference. #9906

ZHUI commented Feb 19, 2025 •

edited

Loading

paddle-bot bot commented Feb 19, 2025

CLAassistant commented Feb 19, 2025

codecov bot commented Feb 19, 2025 •

edited

Loading

DesmonDay Feb 21, 2025

wawltor Feb 27, 2025

wawltor Feb 27, 2025

wawltor Feb 27, 2025

wawltor Feb 27, 2025

wawltor Feb 27, 2025

ZHUI Feb 28, 2025

		return paddle.to_tensor(tensor)


		class EextendDtypeNumpySafe(unittest.TestCase):

		@@ -0,0 +1,226 @@
		# Copyright (c) 2025 PaddlePaddle Authors. All Rights Reserved.

		from .configuration import DeepseekV2Config
		from .fp8_linear import Linear

[Feature] Support float8 dtype storage and deepseek v3 with fp8 inference. #9906

Are you sure you want to change the base?

[Feature] Support float8 dtype storage and deepseek v3 with fp8 inference. #9906

Conversation

ZHUI commented Feb 19, 2025 • edited Loading

Before submitting

PR types

PR changes

Description

paddle-bot bot commented Feb 19, 2025

CLAassistant commented Feb 19, 2025

codecov bot commented Feb 19, 2025 • edited Loading

Codecov Report

DesmonDay Feb 21, 2025

Choose a reason for hiding this comment

wawltor Feb 27, 2025

Choose a reason for hiding this comment

wawltor Feb 27, 2025

Choose a reason for hiding this comment

wawltor Feb 27, 2025

Choose a reason for hiding this comment

wawltor Feb 27, 2025

Choose a reason for hiding this comment

wawltor Feb 27, 2025

Choose a reason for hiding this comment

ZHUI Feb 28, 2025

Choose a reason for hiding this comment

ZHUI commented Feb 19, 2025 •

edited

Loading

codecov bot commented Feb 19, 2025 •

edited

Loading