Help wanted with ANARCII benchmarking

[marinegor]

Hi everyone, on occasion, wanted to say thanks for developers for the awesome tool!

We have recently wrote our own Rust tool for numbering/segmentation, but we're having trouble with correctly benchmarking it against other tools. We're working separately on the correctness benchmarks, but the one I'm really curious is speed. Namely, we run ANARCII on CPU-only machine in CI, but see really weird behavior in benchmarks (see here: https://immunum.enpicom.com/benchmarks/) regardless on where we specify ncpu -- seems that it just doesn't get accounted for. We have an issue tracking that: ENPICOM/immunum#31

Do you have any insights on how to run ANARCII in parallel correctly on CPU machines only?

[ALGW71]

Hi @marinegor - thanks for raising this. It seems to be a torch issue that we have not quite worked out how to fix.

In the code we do the following:

ANARCII/src/anarcii/pipeline/configuration.py

Line 15 in a531cdd

torch.set_num_threads(ncpu)

But it appears that this does not limit torch's use of CPUs. I have a suspicion it may be due to the dataloader. I will get back to you by EOD.

Also - can we have your benchmarking dataset? Those results on your website took terrible for ANARCII!

[marinegor]

@ALGW71 we're running on one of the files from fixtures folder, it's not a private dataset or anything. Sequences there (I recall) are from PDB, and numbering values are those where antpack and ANARCI agree with each other (which is like 95+ percent of all values).

If you know how to fix that, please let me know (or make a PR which I'd appreciate even more!).

My other suspicion was that it can be due to gpu --> cpu transition -- I've seen networks give different results when switching to CPU, especially if some low-level optimizations (e.g. flash-attention) were used for GPU version.

[ALGW71]

@marinegor Can you tell me the exact file in the fixtures folder please.

[marinegor]

Sure, it's this one: https://github.com/ENPICOM/immunum/blob/main/fixtures/validation/ab_H_imgt.csv

Relevant lines: https://github.com/ENPICOM/immunum/blob/05f7f3728d13a1c404a401e1ca6bd145740cbe81/benches/bench_comparison.py#L29-L32

[ALGW71]

@marigor - I have rerun ANARCII on that file and found only one disagreement. See the issue I have opened. Can you take that graph down off your website as it misrepresents ANARCII a lot and from where I am it appears to be a bug in your code somewhere, not a fair representation of the accuracy of ANARCII.

[marinegor]

@ALGW71 thanks a lot for checking that, I think the PR should solve the issue.

As for the performance scaling -- you believe that's solved in #141, right? Did you run any tests for that internally that I can refer to when adapting our internal benchmarks?

[ALGW71]

Hi @marinegor - I just checked locally, but I will not close the PR until you have confirmed you are okay in your tests. My test was just monitoring core usage when requesting 1 core and then all 14 on my Mac ;)

I am just wondering if that will impact CPU speed for the better in my benchmarks, so I can try that and get back to you in a week.

[marinegor]

hi, I've merged a PR with correctness fixes, and ANARCII is indeed now as correct as other tools. The documentation online will be updated with the next release, likely this week! Thanks for bringing that up.

I'm guessing I can now implement changes for the speed benchmarks, right? Is there a particular anarcii version I should use for that?

[ALGW71]

Hi @marinegor - thank you I saw that, looks great!

However RE this PR - I have got some local tests with pytest, and now this new branch causes those to hang (albeit using all cores)... I need to investigate this so cannot merge the PR.

it seems that the code works well for your purpose - 1 cpu. But then when it uses multiple it actually gets slower/hangs...

I will make a note on the PR.

[marinegor]

Ok, no worries. Ping me when you think it's fixed, so that I could update our benchmarks accordingly.