make gensim optional

[helpmefindaname]

Closes #3482

This requires all public models to be in serialized format, so they can be loaded without attempt to load gensim.

Now you can use WordEmbeddings, BytePairEmbeddings in inference without having gensim/bpe installed.

FasttextEmbeddings and MuseEmbeddings will only work when gensim is installed, I find this justifyable, as those embeddings are not commonly used anymore.

When instanciating new WordEmbeddings or BytePairEmbeddings, gensim/bpe is required. You can install them with pip install flair[word-embeddings] after the next release or pip install -e .[word-embeddings] when developing.

[alanakbik]

Thanks a lot for this @helpmefindaname. This generally looks good.

I tested loading our standard 'ner' model that I re-serialized and pushed to the hub as "alanakbik/ner-new".

I then tested loading with the following code:

from flair.data import Sentence
from flair.models import SequenceTagger

model = SequenceTagger.load("alanakbik/ner-new")

sentence = Sentence("Bill was born in New York")

model.predict(sentence)

print(sentence)

That works, but only for newer flair versions. It starts breaking down from Flair 0.10.0.

To reproduce, in a fresh env, do:

pip install flair==0.10.0

and run the above code.

This throws the error:

AttributeError: 'dict' object has no attribute 'embedding_length'

I think this actually has nothing directly to do with this PR, but affects all new models that were trained with newer Flair versions and are loaded with older ones.

But to deploy this PR, we'd need to update all models. So if people are still using an old version of Flair, the regular 'ner' model would no longer work.

@helpmefindaname can you take a look if backward compatibility can be improved?

[alanakbik]

@helpmefindaname thanks a lot for adding this! I will now update all models on HF.

[itsmedonttell]

The pip command recommended by the gensim import error message didn't work out of the box for me. I use Terminal and zsh on macOS. I had to put quotes around "flair[word-embeddings]" when using pip install (like so: pip install "flair[word-embeddings]"), or else zsh throws an error, and the install doesn't execute. Alternatively, pip install genism works.