license of punkt in nltk_data

[happyMindHaha]

Is it possible to use punkt in nltk_data for commercial use freely?
What is the license of punkt in nltk_data?

Thank you.

[stevenbird]

@janstrunk do you have any advice please?

[realchandan]

I have the same question. Some other models clearly mention their licenses. Since it's not explicitly mentioned there, I'd assume it's not open-sourced.

[ykirpichev]

@ekaf I saw you've closed #236.

I would like to let you know that I also have a #239.

Could you please clarify what is the main issues with those two components: punkt in nltk_data ?
Why pull request can't be simply submitted?

[ekaf]

Hi @ykirpichev, let's see how it goes, now that you have submitted your PR.
I think that the authors' initial intention was to opensource Punkt.
But it seems that nobody can or wants to provide a legally binding confirmation.

[ekaf]

It seems very likely that commercial products already exist, that rely on NLTK's Punkt tokenizer,. FWIW, please consider the following snippet generated by Gemini:

It's highly probable that IBM (especially with Watson NLU and related AI products) and Oracle (within their data intelligence and AI offerings), among other large tech companies, do indeed rely on NLTK's Punkt tokenizer. They might not advertise it as a feature, but it serves as a critical underlying component for tasks like sentence segmentation, which is foundational to more complex natural language processing. The licensing disclosures and security patches are strong indicators of this unadvertised, yet essential, integration.

The mentioned security patch Security Bulletin: Vulnerability in Natural Language Toolkit (NLTK)( CVE-2024-39705) affects IBM watsonx Assistant for IBM Cloud Pak for Data does not mention Punkt explicitly, though. But it shows that they were using at least one of the five NLTK data packages that was affected by the vulnerability, and among those Punkt may seem the most likely.

So, as long as the licensing terms of the Punkt data are unclear, it might be worthwhile to look at how its use is acknowledged in commercial products, which have presumably been reviewed by lawyers.