fix: add disallowed_special on tiktoken encode #2102
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
the
size:XS
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
Fixed tiktoken encoding errors by adding disallowed_special=() parameter when processing text containing special tokens like '' during testset generation.
- Modified
ragas/src/ragas/testset/transforms/base.pyto handle special tokens inLLMBasedExtractor.split_text_by_token_limit() - Updated
ragas/src/ragas/utils.pyto add special token handling innum_tokens_from_string()utility - Both changes prevent tiktoken from raising errors when encountering special tokens in documents
2 files reviewed, 1 comment
Edit PR Review Bot Settings | Greptile
Comment on lines
225
to
229
| def num_tokens_from_string(string: str, encoding_name: str = "cl100k_base") -> int: | ||
| """Returns the number of tokens in a text string.""" | ||
| encoding = tiktoken.get_encoding(encoding_name) | ||
| num_tokens = len(encoding.encode(string)) | ||
| num_tokens = len(encoding.encode(string, disallowed_special=())) | ||
| return num_tokens |
There was a problem hiding this comment.
style: Add docstring explaining the disallowed_special parameter and why it's set to empty tuple. This helps future maintainers understand the reasoning behind this configuration.
There was a problem hiding this comment.
@claude add this comment to explain it
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When generating testset, transformation using token count using tiktoken.
but if documents includes special tokens like '' tiktoken raises error.
To prevent it, added parameter "disallowed_special=()" on usages of tiktoken encode