license of punkt in nltk_data

[happyMindHaha]

Is it possible to use punkt in nltk_data for commercial use freely?
What is the license of punkt in nltk_data?

Thank you.

[stevenbird]

@janstrunk do you have any advice please?

[realchandan]

I have the same question. Some other models clearly mention their licenses. Since it's not explicitly mentioned there, I'd assume it's not open-sourced.

[ykirpichev]

@ekaf I saw you've closed #236.

I would like to let you know that I also have a #239.

Could you please clarify what is the main issues with those two components: punkt in nltk_data ?
Why pull request can't be simply submitted?

[ekaf]

Hi @ykirpichev, let's see how it goes, now that you have submitted your PR.
I think that the authors' initial intention was to opensource Punkt.
But it seems that nobody can or wants to provide a legally binding confirmation.

[ekaf]

It seems very likely that commercial products already exist, that rely on NLTK's Punkt tokenizer,. FWIW, please consider the following snippet generated by Gemini:

It's highly probable that IBM (especially with Watson NLU and related AI products) and Oracle (within their data intelligence and AI offerings), among other large tech companies, do indeed rely on NLTK's Punkt tokenizer. They might not advertise it as a feature, but it serves as a critical underlying component for tasks like sentence segmentation, which is foundational to more complex natural language processing. The licensing disclosures and security patches are strong indicators of this unadvertised, yet essential, integration.

The mentioned security patch Security Bulletin: Vulnerability in Natural Language Toolkit (NLTK)( CVE-2024-39705) affects IBM watsonx Assistant for IBM Cloud Pak for Data does not mention Punkt explicitly, though. But it shows that they were using at least one of the five NLTK data packages that was affected by the vulnerability, and among those Punkt may seem the most likely.

So, as long as the licensing terms of the Punkt data are unclear, it might be worthwhile to look at how its use is acknowledged in commercial products, which have presumably been reviewed by lawyers.

[ekaf]

Proposal: Clarifying Punkt's Legal Status and a Path Forward

The legal status of the pre-trained Punkt tokenizer models is currently ambiguous, which prevents their inclusion in a commercial-friendly pip package.

The Core Problem:

• The NLTK code for training the tokenizer is Apache 2.0 licensed.
• The pre-trained model (punkt.pickle) itself lacks an explicit license.
• Crucially, the provenance of the training data is undocumented. It was likely trained on a standard corpus (e.g., Reuters/LDC data) common in academia at the time, which would have restrictive licensing terms.

Because the model is a derivative work of this data, its current license is effectively "unknown." Distributing it without clarity poses a risk.

A Practical Solution: Generate a New, truly Free Punkt Model

Creating a new, unambiguously licensed model is a feasible path forward:

1. Data Source: Train the model on a large corpus of Public Domain texts (e.g., from Project Gutenberg) or texts under a permissive license (e.g., CC0 / CC-BY).
2. Process: Use NLTK's existing Apache 2.0 licensed training code.
3. Result: The resulting model would be a combined work of:
   • Apache 2.0 licensed code
   • Public Domain data
     This allows the entire artifact—the new model—to be confidently released under Apache 2.0.

This approach would resolve the legal ambiguity for Punkt permanently and allow it to be included in a safe nltk-essentials pip package.