Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Support pickle and copy.deepcopy for Sentences #3243

Open
dobbersc opened this issue May 20, 2023 · 0 comments · May be fixed by #3245
Open

[Feature]: Support pickle and copy.deepcopy for Sentences #3243

dobbersc opened this issue May 20, 2023 · 0 comments · May be fixed by #3245
Labels
feature A new feature

Comments

@dobbersc
Copy link
Collaborator

dobbersc commented May 20, 2023 •
edited
Loading

Problem statement

Serializing and deserializing sentences with pickle is a common practice when using Flair in multiprocessing systems, e.g. with the Ray library. Similarly, creating deep copies of objects is a usually natural feature. Unfortunately, loading a pickled sentence with pickle or creating a deep copy with copy.deepcopy fails for sentences with span or relation annotations.

Example:

import copy
import pickle

from flair.data import Sentence

sentence: Sentence = Sentence("Berlin is the capital of Germany.")

# Works: pickle / copy on unannotated sentence
pickled: Sentence = pickle.loads(pickle.dumps(sentence))
copied: Sentence = copy.deepcopy(sentence)
assert pickled.to_original_text() == copied.to_original_text() == "Berlin is the capital of Germany."

# Add span annotations
sentence[:1].add_label("ner", value="LOC")  # Berlin
sentence[5:6].add_label("ner", value="LOC")  # Germany

# Does not work: pickle / copy on span-annotated sentence
# TypeError: __new__() missing 1 required positional argument: 'tokens'
pickled: Sentence = pickle.loads(pickle.dumps(sentence))
copied: Sentence = copy.deepcopy(sentence)
assert (
    {span.text for span in pickled.get_spans("ner")}
    == {span.text for span in copied.get_spans("ner")}
    == {"Berlin", "Germany"}
)

Related to #3191.

Solution

It would be great if sentences with arbitrary annotations were pickable or at least serializable in any manner and copyable.

Additional Context

The reason for the TypeError is sentences annotated with spans or relations contain references to these, where their implementations use the __new__ method.

flair/flair/data.py

Line 550 in 5c53e35

def __new__(self, tokens: List[Token]):

flair/flair/data.py

Line 609 in 5c53e35

def __new__(self, first: Span, second: Span):

To fix this the pickle (and copy) module require implementing a __getnewargs__ or __getnewargs_ex__ method that provides the arguments for __new__ when unpickling an object.
@dobbersc dobbersc added the feature A new feature label May 20, 2023
helpmefindaname added a commit that referenced this issue May 22, 2023
@helpmefindaname
gh-3243: propose a different caching approach for sentence
e870abc
@helpmefindaname helpmefindaname linked a pull request May 22, 2023 that will close this issue
Open
helpmefindaname added a commit that referenced this issue Jul 17, 2023
@helpmefindaname
gh-3243: propose a different caching approach for sentence
a053fcc
helpmefindaname added a commit that referenced this issue Aug 7, 2023
@helpmefindaname
gh-3243: propose a different caching approach for sentence
d64195e
helpmefindaname added a commit that referenced this issue Aug 8, 2023
@helpmefindaname
gh-3243: propose a different caching approach for sentence
22cd5c3
helpmefindaname added a commit that referenced this issue Oct 23, 2023
@helpmefindaname
Merge branch 'master' into gh-3243/add_pickle_support
7021c2e
helpmefindaname added a commit that referenced this issue Mar 29, 2024
@helpmefindaname
Merge branch 'master' into gh-3243/add_pickle_support
a4fd5ed
helpmefindaname added a commit that referenced this issue Aug 30, 2024
@helpmefindaname
gh-3243: propose a different caching approach for sentence
a66a856
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature A new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

gh-3243: support pickle & deepcopy flairNLP/flair
1 participant
@dobbersc