Commit 275dcbf
authored
docs(core): add clarity to base token counting methods (#33958)
Wasn't immediately obvious that `get_num_tokens_from_messages` adds
additional prefixes to represent user roles in conversation, which adds
to the overall token count.
```python
from langchain_google_genai import GoogleGenerativeAI
llm = GoogleGenerativeAI(model="gemini-2.5-flash")
num_tokens = llm.get_num_tokens("Hello, world!")
print(f"Number of tokens: {num_tokens}")
# Number of tokens: 4
```
```python
from langchain.messages import HumanMessage
messages = [HumanMessage(content="Hello, world!")]
num_tokens = llm.get_num_tokens_from_messages(messages)
print(f"Number of tokens: {num_tokens}")
# Number of tokens: 6
```1 parent 9f87b27 commit 275dcbf
1 file changed
+13
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
299 | 299 | | |
300 | 300 | | |
301 | 301 | | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
302 | 305 | | |
303 | 306 | | |
304 | 307 | | |
| |||
317 | 320 | | |
318 | 321 | | |
319 | 322 | | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
320 | 326 | | |
321 | | - | |
322 | | - | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
323 | 334 | | |
324 | 335 | | |
325 | 336 | | |
| |||
0 commit comments