selected_with_duplicates don't return all objects when there is an exact match

[Lhemamou]

selected_with_duplicates return only one object when there is an exact match.

from semhash import SemHash
create a test subset

test_records = [
{"name": "Adhesion", "idx": 0},
{"name": "Adhesion", "idx": 1},
{"name": "Adhesion", "idx": 2},
{"name": "Adhesion", "idx": 3},
{"name": "Adhesion", "idx": 4},
{"name": "Adhesion", "idx": 5},
{"name": "Adhesion", "idx": 6},
{"name": "Adhesion", "idx": 7},
{"name": "Adhesion", "idx": 8},
{"name": "Adhesion", "idx": 9},
]

semhash_test = SemHash.from_records(records=test_records, columns=["name"])

deduplicated_texts = semhash_test.self_deduplicate()
select_with_dup = deduplicated_texts.selected_with_duplicates
print(select_with_dup)

print(select_with_dup)
[SelectedWithDuplicates(record={'name': 'Adhesion', 'idx': 0}, duplicates=[({'name': 'Adhesion', 'idx': 9}, 1.0)])]

I suspect the hash function to work only on name and not on the complete record.

[Pringled]

Hi @Lhemamou, thanks for reporting, great catch. I've added a fix for this in this PR: #71. I'll do a bit more testing this evening, but I think this fixes it. If you want you can already try it on that branch, when I run your code I now get:

[SelectedWithDuplicates(record={'name': 'Adhesion', 'idx': 0}, duplicates=[({'name': 'Adhesion', 'idx': 1}, 1.0), ({'name': 'Adhesion', 'idx': 2}, 1.0), ({'name': 'Adhesion', 'idx': 3}, 1.0), ({'name': 'Adhesion', 'idx': 4}, 1.0), ({'name': 'Adhesion', 'idx': 5}, 1.0), ({'name': 'Adhesion', 'idx': 6}, 1.0), ({'name': 'Adhesion', 'idx': 7}, 1.0), ({'name': 'Adhesion', 'idx': 8}, 1.0), ({'name': 'Adhesion', 'idx': 9}, 1.0)])]

[Lhemamou]

It looks good ! Thanks for the PR !

[Pringled]

Perfect, just released this: https://github.com/MinishLab/semhash/releases/tag/v0.3.3. If you install the latest version this is now resolved. Thanks again for reporting!

[Lhemamou]

Thanks to you for your work !