Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] ONNX conversion #6

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
Open

[WIP] ONNX conversion #6

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
498db46
Replace with Dropout and Softmax
ganik Jul 20, 2020
3be8289
mask attention scores in Softmax
ganik Jul 21, 2020
dab83af
onnx conversion and training
ganik Aug 2, 2020
64f068c
TBD tight coupling with torch 1.3
ganik Aug 2, 2020
1ce5cc1
opset 12, expand attention mask
ganik Aug 7, 2020
efd079b
Merge
ganik Aug 7, 2020
b301b46
loss is first
ganik Aug 8, 2020
d2fa9fd
commenting out v_ and q_ biases as they are always const
ganik Aug 8, 2020
e4793b8
Fix Dropout model regression issue
ganik Aug 14, 2020
155d966
Use nn.dropout and nn.softmax by default
ganik Aug 14, 2020
95ec7ad
Added ORT Glue based tests
Sep 29, 2020
bf8a3ce
remove onnx path in train.py
Sep 29, 2020
c71818b
Add Readme
Sep 29, 2020
7eff1fd
Use random seed by default
Sep 29, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion DeBERTa/deberta/disentangled_attention.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,9 @@ def linear(w,b,x):
if self.talking_head:
attention_scores = self.head_logits_proj(attention_scores.permute(0,2,3,1)).permute(0,3,1,2)

attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1)
#attention_probs = XSoftmax.apply(attention_scores, attention_mask, -1)
nodex = torch.nn.Softmax(-1)
ganik marked this conversation as resolved.
Show resolved Hide resolved
attention_probs = nodex(attention_scores)
ganik marked this conversation as resolved.
Show resolved Hide resolved
attention_probs = self.dropout(attention_probs)
if self.talking_head:
attention_probs = self.head_weights_proj(attention_probs.permute(0,2,3,1)).permute(0,3,1,2)
Expand Down
6 changes: 5 additions & 1 deletion DeBERTa/deberta/ops.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,11 @@ def backward(ctx, grad_output):
else:
return grad_output, None

class StableDropout(torch.nn.Module):
class StableDropout(torch.nn.Dropout):
ganik marked this conversation as resolved.
Show resolved Hide resolved
def __init__(self, drop_prob):
super().__init__()

class StableDropout1(torch.nn.Module):
""" Optimized dropout module for stabilizing the training

Args:
Expand Down