Skip to content

README.md

Latest commit

 

History

History
56 lines (39 loc) · 1.14 KB

README.md

File metadata and controls

56 lines (39 loc) · 1.14 KB

tiktoken-cli

Count OpenAI tokens from the command line. Useful for estimating API costs and staying within context limits.

Install

pip install tiktoken
curl -o tiktoken_cli.py https://raw.githubusercontent.com/himalaya0x/tiktoken-cli/master/tiktoken_cli.py

Usage

# Count tokens in a string
python3 tiktoken_cli.py "Hello, world!"

# Count tokens in a file
python3 tiktoken_cli.py -f prompt.txt

# Specify model encoding
python3 tiktoken_cli.py -m gpt-4 "What is the meaning of life?"

# Read from stdin
cat document.txt | python3 tiktoken_cli.py

# Show token IDs
python3 tiktoken_cli.py --ids "Hello, world!"

# Compare across encodings
python3 tiktoken_cli.py --compare "Hello, world!"

Output

$ python3 tiktoken_cli.py -m gpt-4 "What is the meaning of life?"
Model:    gpt-4
Encoding: cl100k_base
Tokens:   7
Chars:    29
Ratio:    4.1 chars/token

Supported Models

All OpenAI model encodings via tiktoken:

  • gpt-4 / gpt-4-turbo / gpt-4o → cl100k_base (default)
  • gpt-3.5-turbo → cl100k_base
  • text-davinci-003 → p50k_base
  • text-embedding-ada-002 → cl100k_base

License

MIT