ModuleNotFoundError: No module named 'spacy.lang.tokenizer'

[LeJudith]

Hi again!

I successfully migrated the pretrained a MedCAT v1 UMLS Dutch (v1.10) model into MedCAT v2, following the official migration notebook.
The base migrated model loads and works perfectly fine.

However, after fine-tuning it using annotations from MedCATTrainer and saving the model pack, I can no longer reload it.
When I attempt to load the saved fine-tuned model pack, I get this error:

ModuleNotFoundError: No module named 'spacy.lang.tokenizer'

The above exception was the direct cause of the following exception:

ImportError                               Traceback (most recent call last)
Cell In[5], [line 1](vscode-notebook-cell:?execution_count=5&line=1)
...
--> [387](https://vscode-remote+ssh-002dremote-002boaks-002ddebug.vscode-resource.vscode-cdn.net/home/user/venvs/prism/lib/python3.10/site-packages/spacy/util.py:387)             raise ImportError(Errors.E048.format(lang=lang, err=err)) from err
    388     set_lang_class(lang, getattr(module, module.__all__[0]))  # type: ignore[attr-defined]
    389 return registry.languages.get(lang)

ImportError: [E048] Can't import language tokenizer or any matching language from spacy.lang: No module named 'spacy.lang.tokenizer'

This issue only occurs with the Dutch model , not with English SNOMED models fine-tuned in the exact same way.

Is this a known issue and is there a workaround for this?

Thanks already in advance!

[mart-r]

Hi!

Thanks for bringing this to our attention! It seems there is indeed somethign wrong with how we handle the file loading in this case. I've been able to replicate the error, but I'm not yet 100% sure what the underlying issue is.

I will let you know if I find out and what you may be able to do for a workaround if one exists as well as implement a general fix for future releases.

[mart-r]

Hi again!

I've identified the issue and implemented the fix in #213 .

This has to do with how models are loaded off disk, so my best suggestion (until we get the fix released) is to simply load the v1 model (i.e convert on the fly) instead of saving it on disk and loading from there. It'll make model loading quite a bit slower, but at least it'll be successful.

Sorry for the inconvenience! And thanks for bringing this to our attention!

PS:
After the fix in #213 is merged in, ne models will be saved in the correct format. But older models will also be able to be loaded correctly (even if there's an issue with the way they were saved).

[LeJudith]

Hi! Thanks so much for identifying and fixing the issue!

I actually noticed this problem when saving and reloading the fine-tuned model, rather than during conversion. For now, fine-tuning the model and directly using it (on the fly) without saving/re-loading it indeed works.

Do you happen to have an estimate on when this fix might be merged?

Thanks again for your help!

[mart-r]

Unfortunately I can't give you too accurate an estimation. However, I would hope that it'd be merged in some time this week. And if all goes well I can subsequently push a patch release.

[mart-r]

@LeJudith
We've now released MedCAT v2.3.0 that should fix this issue along with a few others.
You can install from PyPI as you did v2.2.0.

Let us know if you have any further issues.

[LeJudith]

Thanks, I just tested it and can confirm that it works!