Enforce locale for fake entities

[julcsii]

I would like to ask how to make sure that the generated fake entities are corresponding to the language/locale I want to use.

Small example:

from presidio_evaluator.data_generator import PresidioSentenceFaker

sentence_templates = [
    "{{name}} ist in München geboren.",
    "Bitte die Rechnung an {{address}} schicken!"
]
sentence_faker = PresidioSentenceFaker(
    locale="de_DE",
    lower_case_ratio=lower_case_ratio,
    sentence_templates=sentence_templates,
    random_seed=42,
    provider_aliases=PresidioSentenceFaker.PROVIDER_ALIASES
)

fake_records = sentence_faker.generate_new_fake_sentences(num_samples=number_of_samples)
pprint.pprint(fake_records[1])

The result:

Full text: Bitte die Rechnung an 32181 Tuulimyllyntie 27 Suite 001 HIRVENSALMI Southern Savonia schicken!
Spans: [Span(type: STREET_ADDRESS, value: 32181 Tuulimyllyntie 27 Suite 001 HIRVENSALMI Southern Savonia, char_span: [22: 84])]

There are also often German/Austrian names and addresses, but not only. Is there a way to somehow force to only generate entities that correspond to the locale I specify?

[omri374]

Thanks for raising this. The reason this is happening is because PresidioSentenceFaker loads a data frame to augment Faker's values with more data, but it's not locale specific. I'll leave the issue open and we'll address this. A workaround would be to replace the SentenceFaker object used under the hood:

sentence_faker = PresidioSentenceFaker(
    locale="de_DE",
    lower_case_ratio=lower_case_ratio,
    sentence_templates=sentence_templates,
    random_seed=42,
    provider_aliases=PresidioSentenceFaker.PROVIDER_ALIASES
)


from presidio_evaluator.data_generation.faker_extensions import SentenceFaker
# workaround
sentence_faker._sentence_faker(locale="de_DE", lower_case_ratio=lower_case_ratio)

This would prevent the underlying faker from leveraging the fake data frame.

[julcsii]

Thank you for the quick response!

After applying your suggestion I see only German entities, which is great. But this also means that a few entity types I want to generate, I cannot.

When do you plan to address this and do you have some ideas on how?

[omri374]

Option 1 is to create a similar pandas DataFrame as the default one, just with German names / PII entities. We've used https://www.fakenamegenerator.com/order.php

Option 2 is to create your own Faker providers. For reference, take a look at the providers created here:
https://github.com/microsoft/presidio-research/blob/master/presidio_evaluator/data_generator/faker_extensions/providers.py

Then, you can pass those providers into the PresidioSentenceFaker object, like this:

import random
from faker.providers import BaseProvider


class MarsIdProvider(BaseProvider):
    def mars_id(self):
        # Generate a random row number between 1 and 50
        row = random.randint(1, 50)
        # Generate a random letter for the seat location from A-K
        location = random.choice("ABCDEFGHIJK")
        # Return the seat in the format "row-letter" (e.g., "25A")
        return f"{row}{location}"


sentence_faker.add_provider(MarsIdProvider)
# Now a new `mars_id` entity can be generated if a template has `mars_id` in it.

Option 3, is to see if Faker already supports the entities you're interested in, but with a different name. In this case, you could use the provider_alias dictionary to create a mapping between the entities in the templates dataset, and the entities that Faker can produce. A simple example is the default ones:

provider_aliases = [
    ("name", "person"),
    ("credit_card_number", "credit_card"),
    ("date_of_birth", "birthday"),
]