Fix NameError in extract_entities_long; collapse duplicate surfaces when spans are stripped - #137
Open
blockems wants to merge 2 commits into
Open
Conversation
extract_entities_long forwards `overlap_policy=overlap_policy` to extract_long
but never declares the parameter, so every call raises
NameError: name 'overlap_policy' is not defined
The sibling wrappers thread it correctly — extract_long and batch_extract_long
both declare `overlap_policy: Optional[str] = None` — so this looks like one
wrapper missed when the parameter was added. batch_extract_entities_long is
unaffected because it does not forward the name.
Reproduces on any call:
from gliner2 import GLiNER2
model = GLiNER2.from_pretrained("fastino/gliner2-base-v1")
model.extract_entities_long("some long document ...", ["person"])
Fix adds the parameter to the signature, matching extract_long's default.
_dedupe_items deduplicates spans by position, which correctly keeps the same
surface text at different offsets. But when include_spans and include_confidence
are both false, _strip_span_metadata reduces those spans to bare strings — and
distinct positions then become identical strings no caller can tell apart.
On a long document this is very visible. A report mentioning "PostgreSQL" 300
times returns it 300 times:
{"entities": {"database": ["PostgreSQL", "PostgreSQL", ... x300]}}
The repeats carry no information at that fidelity: there is no offset to
distinguish them and no confidence to rank them, so a caller can only discard
them. Collapse them, preserving first-seen document order.
Scoped to lists that reduce entirely to strings, so anything still carrying
confidence, offsets or attribute-group payloads is untouched. Verified against
fastino/gliner2-base-v1 on a document repeating one sentence 120 times:
formatted: database 3 unique, location 1 unique
include_spans=True: 49 entries, all distinct positions preserved
tests/inference and tests/processing pass (71 tests).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent fixes to the long-document path, found while serving GLiNER2 behind an HTTP API. Happy to split into separate PRs if you prefer.
1.
extract_entities_longraisesNameErroron every callextract_entities_longforwardsoverlap_policy=overlap_policytoextract_long, but its own signature never declares the parameter:extract_longandbatch_extract_longboth declareoverlap_policy: Optional[str] = None, so this looks like one wrapper missed when the parameter was threaded through.batch_extract_entities_longis unaffected because it does not forward the name — which is why the batch path works and the single-text path does not.Fix adds the missing declaration, matching
extract_long's default.2. Duplicate surfaces once spans are stripped
_dedupe_itemsdeduplicates spans by position, which correctly keeps the same surface text at different offsets. Wheninclude_spansandinclude_confidenceare both false,_strip_span_metadatathen reduces those spans to bare strings — and the distinct positions become identical strings that no caller can tell apart.On a long document this is very visible. A report mentioning "PostgreSQL" 300 times returns it 300 times:
{"entities": {"database": ["PostgreSQL", "PostgreSQL", ...x300]}}At that fidelity the repeats carry no information — no offset to distinguish them, no confidence to rank them — so a caller can only discard them.
The fix collapses them while preserving first-seen document order, scoped to lists that reduce entirely to strings. Anything still carrying confidence, offsets or attribute-group payloads is untouched, so
include_spans=Truebehaviour is unchanged.Verified against
fastino/gliner2-base-v1on a document repeating one sentence 120 times:database: ~200 entriesdatabase: 3 unique,location: 1 uniqueinclude_spans=TrueI read the note in the long-document tutorial about keeping distinct mentions at different document positions — that behaviour is preserved wherever positions are actually visible in the output. If you would rather have this behind an opt-in flag than as a default, say the word and I will rework it.
Testing
tests/inferenceandtests/processingpass — 71 tests.