AMPLIFY Mega-PR

[pstjohn]

No description provided.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 90.65421% with 20 lines in your changes missing coverage. Please review.

Project coverage is 86.37%. Comparing base (b3829e0) to head (1542641).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...ges/bionemo-amplify/src/bionemo/amplify/convert.py	73.84%	17 Missing ⚠️
...kages/bionemo-amplify/src/bionemo/amplify/model.py	97.52%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #640      +/-   ##
==========================================
+ Coverage   86.27%   86.37%   +0.10%     
==========================================
  Files         118      121       +3     
  Lines        7162     7363     +201     
==========================================
+ Hits         6179     6360     +181     
- Misses        983     1003      +20     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pstjohn]

So yeah, I have really no idea what to make of this. The accuracy starts degrading steadily through each attention block; this test fails after the second block with an error of ~0.01.

I tried using a non-TE attention block, as well as turning off all the fusions with no luck. Some other things to try would be omitting the mask tokens in the test inputs and see if that helps figure out where this is coming from.