Mind if the OpenAI Cookbook repo points to your Tiktokenizer app?

[ted-at-openai]

I got a PR suggesting to add a link to your Tiktokenizer app from our OpenAI tokenization tutorial: openai/openai-cookbook#604

Would this be ok with you?

I don't want to send lots of traffic your way if:

• it costs you $
• you're planning to shutdown/modify soon
• it makes you feel obligated to support
• or anything like that

Cheers,
Ted

[nicognaW]

No offense, but why is OpenAI not updating the official tokenizer https://platform.openai.com/tokenizer to the latest version and instead wants to use community work?

[ted-at-openai]

No offense taken. We'll update our version at some point, but no one on our team has prioritized it. In the meantime, an appreciative developer suggested we add a link to this version. Happy to link or not. My goal is to be helpful, not take advantage of community work.

[dqbd]

Hi @ted-at-openai!
Sounds fine by me, glad that the tool is useful and helpful to this day 🙂

---

In general the end-goal for this tool is to merge the functionality into the official tokenizer page, at which point I can redirect to the official tokenizer instead. Let me know if I can help with that 🙂.