longlora-paddle #9939
longlora-paddle #9939
Conversation
|
Thanks for your contribution! |
|
|
Codecov ReportAttention: Patch coverage is
❌ Your project check has failed because the head coverage (17.36%) is below the target coverage (58.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## develop #9939 +/- ##
============================================
- Coverage 51.33% 17.36% -33.98%
============================================
Files 745 746 +1
Lines 118600 120302 +1702
============================================
- Hits 60886 20888 -39998
- Misses 57714 99414 +41700 ☔ View full report in Codecov by Sentry. |
| @@ -213,6 +213,111 @@ def parallel_matmul(x: Tensor, y: Tensor, transpose_y=False, tensor_parallel_out | |||
| return logits | |||
|
|
|||
|
|
|||
| def shift(qkv, bsz, q_len, group_size, num_heads, head_dim): | |||
There was a problem hiding this comment.
不要把longlora这部分写到llama的组网里面，可以考虑写到paddlenlp/transformers/longlora.py中，如果使用longlora那么调用replace函数，现在llama组网已经太复杂了，不要再往这里写了。https://github.com/dvlab-research/LongLoRA/blob/main/llama_attn_replace.py#L460
|
|
||
| def run_predictor(self, config_params=None): | ||
| if config_params is None: | ||
| config_params = {} |
There was a problem hiding this comment.
test longlora是不是可以直接写一个测试longlora函数是否能够跑通并且结果符合预期的测试就好，本身已经有一个longlora训练的测试了
PR types
New features
PR changes
Models
Description
Paddle implementation of LongLoRA