Finally got tiktoken luarock working with Windows and PowerShell - How To Guide

[GitMurf]

Overview

I will submit a PR to add to the README once I have vetted this fully out, get some feedback and get confirmation from someone else they are able to get tiktoken working on Windows using these instructions. I don't want to forget any of the steps while they are fresh in my mind!

Gotchas

Modify the suggested luarocks install command from the README

README location to amend

Currently the README has some incorrect info for Windows / PowerShell users with this documented command: sudo luarocks install --lua-version 5.1 tiktoken_core

Corrections to work with Windows:

1. Remove the word sudo as that is not a Windows / PowerShell thing.
2. Need to use an = equals instead of a space between --lua-version 5.1
3. Here is the final working command: luarocks install --lua-version=5.1 tiktoken_core
4. Readme mentions Save the binary as tiktoken_core.so in any of the given paths which confused me for a bit because on Windows it is .dll instead of .so (see next section below).

Confusion with tiktoken_core.so vs tiktoken_core.dll

In my reading and research on luarocks and the ecosystem, I saw a ton about *.so files but hardly anything about *.dll files. This is because *.dll is the Windows equivalent of *.so on Mac / Linux. So you can think of them as synonymous.

The initial Lua for Windows all-in-one package has older version of luarocks

As you see in steps 2 - 8 below, it took me a while to realize that while you need to use the "all-in-one package" for the initial installation of Lua / LuaRocks for Windows, you then need to "update" luarocks by downloading the newest win32 binary (step 2 below) and place it in the Lua 5.1 install directory (step 6 below) so that the newest version of luarocks is used instead of the default old one.

Step by Step

1. Install Lua / LuaRocks with the Lua for Windows all-in-one package here: https://github.com/rjpcomputing/luaforwindows/releases/tag/v5.1.5-52
2. Download the latest version of LuaRocks 3.11.1 from here: https://luarocks.github.io/luarocks/releases/
   • Make sure to select the win32 version (NOT win64) of the stand-alone luarocks.exe binary which should be titled: luarocks-3.11.1-windows-32.zip
   • Here is the direct download link to make sure you are selecting the correct one in case you are unsure: https://luarocks.github.io/luarocks/releases/luarocks-3.11.1-windows-32.zip
3. Extract the downloaded zip
4. Copy luarocks.exe and luarocks-admin.exe files from the extracted zip to your Lua / LuaRocks install directory which should be something like this: C:\Program Files (x86)\Lua\5.1\
   • TIP: if you have where.exe command line tool installed, you can use where.exe luarocks to find the path.
5. Restart PowerShell to make sure everything is refreshed and ready to use.
6. Validate luarocks is now updated to v3.11.1 by running luarocks --version
7. Now install tiktoken with: luarocks install --lua-version=5.1 tiktoken_core
   • It will likely warn that you are missing the dependency luarocks-build-rust-mlua but should auto install that for you as well.
   • If for some reason it does not auto install luarocks-build-rust-mlua for you, go ahead and install it separately with luarocks install --lua-version=5.1 luarocks-build-rust-mlua
8. It should install tiktoken to somewhere like: C:\Users\USER_NAME\AppData\Roaming\luarocks\
9. Again close and re-open PowerShell to make sure the installs are all loaded.
10. Finally you need to tell Neovim where to look for luarock packages by setting the environment variable LUA_CPATH (mine was NOT set by default).
    • See below code block for what I added to my PowerShell PROFILE which loads things like environment variables before every PowerShell session.
    • Run the following luarock command to get the correct cpath and set the environment variable accordingly: luarocks path --lr-cpath.

# set to result of `luarocks path --lr-cpath`
$Env:LUA_CPATH="C:\Users\USER_NAME\AppData\Roaming\luarocks\lib\lua\5.1\?.dll"

That is it! Now if you restart PowerShell and startup Neovim, tiktoken should be working. You can validate it is by running the health check: checkhealth CopilotChat.