Merge pull request openai#281 from openai/ted/update-verbiage-on-tokenization

ted-at-openai · web-flow · commit ebfdfe30998a · 2023-03-25T00:15:15.000-07:00
updates description of token counting function
diff --git a/examples/How_to_count_tokens_with_tiktoken.ipynb b/examples/How_to_count_tokens_with_tiktoken.ipynb
@@ -434,7 +434,7 @@
     "\n",
     "Below is an example function for counting tokens for messages passed to `gpt-3.5-turbo-0301` or `gpt-4-0314`.\n",
     "\n",
-    "Note that the exact way that messages are converted into tokens may change from model to model, and may even change over time for the same model. Therefore, the counts returned by the function below should be considered an estimate, not a guarantee."
+    "Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee."
    ]
   },
   {
diff --git a/examples/How_to_format_inputs_to_ChatGPT_models.ipynb b/examples/How_to_format_inputs_to_ChatGPT_models.ipynb
@@ -437,7 +437,7 @@
     "\n",
     "You can use the following function to count the number of tokens that a list of messages will use.\n",
     "\n",
-    "Note that the exact way that messages are converted into tokens may change from model to model, and may even change over time for the same model. Therefore, the counts returned by the function below should be considered an estimate, not a guarantee.\n",
+    "Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee.\n",
     "\n",
     "Read more about counting tokens in [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb)."
    ]

Original file line number	Diff line number	Diff line change
`@@ -434,7 +434,7 @@`
`434`	`434`	`"\n",`
`435`	`435`	"Below is an example function for counting tokens for messages passed to `gpt-3.5-turbo-0301` or `gpt-4-0314`.\n",
`436`	`436`	`"\n",`
`437`		`- "Note that the exact way that messages are converted into tokens may change from model to model, and may even change over time for the same model. Therefore, the counts returned by the function below should be considered an estimate, not a guarantee."`
	`437`	`+ "Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee."`
`438`	`438`	`]`
`439`	`439`	`},`
`440`	`440`	`{`
Original file line number	Diff line number	Diff line change
`@@ -437,7 +437,7 @@`
`437`	`437`	`"\n",`
`438`	`438`	`"You can use the following function to count the number of tokens that a list of messages will use.\n",`
`439`	`439`	`"\n",`
`440`		`- "Note that the exact way that messages are converted into tokens may change from model to model, and may even change over time for the same model. Therefore, the counts returned by the function below should be considered an estimate, not a guarantee.\n",`
	`440`	`+ "Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee.\n",`
`441`	`441`	`"\n",`
`442`	`442`	`"Read more about counting tokens in [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb)."`
`443`	`443`	`]`