Improve score by supporting extra_phrase for extra words in rules

[alok1304]

Follow up of:

• Improve score by supporting extra_phrase for extra words in rules #4424

Add new phrases like extra_phrase this is special for extra-words. This phrase is represented in the format [[n]], where n indicates the maximum number of extra-words allowed at that position in the rule.

If extra-words appear at the correct position and their count does not exceed the allowed limit n, then the score is increased to 100.

Reference #4420

Tasks

• Reviewed contribution guidelines
• PR is descriptively titled 📑 and links the original issue above 🔗
• Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
  Run tests locally to check for errors.
• Commits are in uniquely-named feature branch and has no merge conflicts 📁
• Updated documentation pages (if applicable)
• Updated CHANGELOG.rst (if applicable)

Signed-off-by: Alok Kumar [email protected]

[AyanSinhaMahapatra]

Thanks @alok1304! Looking much better

See comments for your consideration. I've updated your PR description to mention that this is a follow up PR, since there is important context and reviews in the previous PR, we need to preserve this as required.

[alok1304]

I addet test for 3-seq where there is no detection of copyrights statements , Ref: https://github.com/xyzzy-022/xyzzy/blob/5a16eb998470241b33ad3caa6a4946d0448a16b6/LEGAL.md?plain=1#L97
this file when we scan we got extra-words so I added extra-phrase marker in that corresponding matched rule. Such that we can improve the score.

[alok1304]

Actually, the previous test case is failing because of extra-phrase is removed from rules while loading, after that when scan same rules then matcher is 3-seq not any one of these"1-hash", "1-spdx-id", "2-aho" this is due to extra-phrase marker that are present in rules. This test case is failing https://github.com/aboutcode-org/scancode-toolkit/blob/develop/tests/licensedcode/test_detection_validate.py#L101

   def check_rule_or_license_can_be_detected_exactly(licensish):
        """
        Check that a rule or license can be detected exactly, either by thyself (or with an exact match
        to any rule for deprecated rules).
        """
        idx = cache.get_index()
        deadline = time() + 20  # ms
        matches = idx.match(query_string=licensish.text, _skip_hash_match=True, deadline=deadline)
        # ensure we can self-detect exactly
        expected = [licensish.identifier]
        results = [m.rule.identifier for m in matches]
    
        if results != expected:
            expected.append(f'file://{licensish.rule_file()}')
            assert results == expected
        icm = is_correct_detection(matches)
        if not icm:
            expected.append(f'file://{licensish.rule_file()}')
>           assert results == expected

E           AssertionError: assert ['bsd-new_newlib3.RULE'] == ['bsd-new_newlib3.RULE',\n 'file:///home/vsts/work/1/s/src/licensedcode/data/rules/bsd-new_newlib3.RULE']
E             Right contains one more item: 'file:///home/vsts/work/1/s/src/licensedcode/data/rules/bsd-new_newlib3.RULE'
E             Full diff:
E               [
E                'bsd-new_newlib3.RULE',
E             -  'file:///home/vsts/work/1/s/src/licensedcode/data/rules/bsd-new_newlib3.RULE',
E               ]

because is_correct_detection gives False because it not include 3-seq in matcher see: https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/detection.py#L1090
That's why the test case is failing. So, I added 3-seq see: https://github.com/aboutcode-org/scancode-toolkit/pull/4432/files#diff-5f6ccc1d8f9540cae715bddaf337246f4c69c48ca596b2086e05b18c541aef4aR1098

Now, all the test cases are passed :)

[AyanSinhaMahapatra]

Thanks @alok1304. looking great overall, see comments for your consideration.

[AyanSinhaMahapatra]

Thanks @alok1304, looking good!

Two minor changes requested for your consideration:

1. in the test and test filenames we should add the new case as a new test and let the old tests (which we have perfect scores for now) stay as-is
2. In the rule detection tests we should process the text before running license detection instead of ignoring rules with extra word markers.

Asking a review from @pombredanne too as this is ready for him to take a look at.

[AyanSinhaMahapatra]

Can you also use the --copyright option here so we have copyright detection in the resource, we have to use this to resolve the issue.

[alok1304]

Hey @AyanSinhaMahapatra, this is ready to merge.

[pombredanne]

I wonder if the scoring should not be more subtle. In practice, we can have multiple regions allowing for extra words. And some regions may have OK extra words, and some other may not have that, so this is may not be a binary True or False, with 100 or not 100 but instead a score computation that may be more nuanced?