wikipedia-mixer.json: question about exclude field

[Nevermetyou65]

Hi

This might not related to dolma, but I don't understand the jsonpath in this file
https://github.com/allenai/dolma/blob/main/docs/examples/wikipedia-mixer.json

Specifically,

"exclude": [
            "$.attributes[?(@.exp__whitespace_tokenizer_with_paragraphs_v1__document[0][2] < 50)]",
            "$.attributes[?(@.exp__ft_lang_id_en_paragraph_with_doc_score_v2__doc_en[0][2] <= 0.5)]",
            "[email protected][?(@.bff_duplicate_paragraph_spans && @.bff_duplicate_paragraph_spans[0] && @.bff_duplicate_paragraph_spans[0][2] >= 1.0)]"
          ]

I don't understand the 3rd element, why is it "[email protected]"? and not just "$.attributes" like the twos above?

[aman-17]

Hey @Nevermetyou65 this is likely a typo, since @ isn’t typically used at the root level. It should probably be $.attributes[...], just like the others.

[Nevermetyou65]

Thanks for your answer. Are you sure this is a typo? Actually, I think this a typo too but in

• https://github.com/allenai/dolma/blob/main/configs/dolma-v1_5/mixing/cc-head.yaml
• https://github.com/allenai/dolma/blob/main/configs/dolma-v1_6/mixing/cc-head.yaml

[email protected] is used. And I see it is used only with bloom filter. Really don't know why though.

[aman-17]

I need to test this myself to see if it’s a custom extension or syntax specific to the framework. If it has a special meaning in this context that differs from standard JSONPath interpretation, then it would be correct.

[Nevermetyou65]

If that the case, I think a documentation would be very much prefer. Dolma doc is kind of lacking to me. Sorry to say this.

[aman-17]

Appreciate your feedback.

[Nevermetyou65]

Thanks for your response!!