Established way of getting duplicate ids?

[AltamashRafiq]

Hi! First of all, thank you for the amazing set of packages! My team and I are big fans of your work 😄

I am wondering if there is an established way to get duplicate ids. Lets take the code below:

from semhash import SemHash

data = [
    {"id": 0, "text": "hello"},
    {"id": 1, "text": "hello"},
    {"id": 2, "text": "helllo"},
]

semhash = SemHash.from_records(records=data, columns=['text'])
deduplicator = semhash.self_deduplicate(threshold=0.0000001)

This gives for deduplicator.selected:

[{'id': 0, 'text': 'hello'}]

And for deduplicator.filtered:

[DuplicateRecord(record={'id': 1, 'text': 'hello'}, exact=True, duplicates=[({'id': 0, 'text': 'hello'}, 1.0)]),
 DuplicateRecord(record={'id': 2, 'text': 'helllo'}, exact=False, duplicates=[({'id': 0, 'text': 'hello'}, 0.5699238181114197), ({'id': 1, 'text': 'hello'}, 0.5699238181114197)])]

It is nice to get the duplicate records from the filtered attribute but it would also be great to have the reverse of this. I.e. for each record in deduplicator.selected, it would be nice to have a list of the duplicates. To my knowledge, the way to get this using the current implementation is to loop over the filtered for each item in selected. Would there be an easier / already established way of doing this?

It would be cool if we could (hypothetically) get for deduplicator.selected:

[DeDuplicatedRecord(record={'id': 0, 'text': 'hello'}, duplicates=[<duplicates_here>])]

[Pringled]

Hi @AltamashRafiq, thanks for the kind words, happy to hear that!

I like this idea; I'm looking into whether we can add some native functionality for this in SemHash at the moment. I'll keep you posted on the progress here.

[AltamashRafiq]

@Pringled that's awesome, thank you!

[Pringled]

Hi @AltamashRafiq, we just merged a PR that adds what you need: #64. We also just did a new release, so if you upgrade to the latest version you can now do the following:

from semhash import SemHash

data = [
    {"id": 0, "text": "hello"},
    {"id": 1, "text": "hello"},
    {"id": 2, "text": "helllo"},
]

semhash = SemHash.from_records(records=data, columns=['text'])
deduplicator = semhash.self_deduplicate(threshold=0.0000001)
print(deduplicator.selected_with_duplicates)

This will give you the following:
[SelectedWithDuplicates(record={'id': 0, 'text': 'hello'}, duplicates=[({'id': 1, 'text': 'hello'}, 1.0), ({'id': 2, 'text': 'helllo'}, 0.10063767433166504)])] (for every record you get the duplicates and scores, like in your example).

Let me know if everything works as expected!

[AltamashRafiq]

@Pringled that's awesome! I'll give it a try later today.

[AltamashRafiq]

@Pringled I tried it - it works great! Thanks for taking the time to address the need!

[Pringled]

@AltamashRafiq Happy to hear that! Will close this, but feel free to open another issue if you have more ideas/requests!