Skip to content

new tokenizer system support#257

Closed
dimapihtar wants to merge 23 commits intomainfrom
dpykhtar/new_tokenizer_system
Closed

new tokenizer system support#257
dimapihtar wants to merge 23 commits intomainfrom
dpykhtar/new_tokenizer_system

Conversation

@dimapihtar
Copy link
Contributor

@dimapihtar dimapihtar commented Jul 16, 2025

No description provided.

@dimapihtar
switch to new tokenizer system
a752408 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@copy-pr-bot
Copy link

copy-pr-bot bot commented Jul 16, 2025

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

dimapihtar and others added 13 commits July 16, 2025 06:35
@dimapihtar
fix stlye
ce7ea22 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
fix style
ec7a87f 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
fix style
702b511 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
fix import
42a0ea5 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
remove extra args
4595f0a 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
minor fix
0ba0c26 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
add tokenizer unit tests
a74b245 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
Merge branch 'main' into dpykhtar/new_tokenizer_system
b76f094 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
fix imports
ee85bbc 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
Merge branch 'main' into dpykhtar/new_tokenizer_system
b143260
@dimapihtar
update to main
b8b77cd 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
Merge branch 'main' into dpykhtar/new_tokenizer_system
e2a0d75
@dimapihtar
remove extra files
666b460 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar dimapihtar marked this pull request as ready for review August 9, 2025 18:35
@dimapihtar dimapihtar marked this pull request as draft August 9, 2025 18:37
dimapihtar and others added 9 commits August 9, 2025 11:44
@dimapihtar
fix style
4e73381 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
fix style
638008a 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
remove old tokenizers
ca269c4 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
minor changes
9a1a038 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
fix tokenizers
689b6cd 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
fixt tests
00d19e8 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
Merge branch 'main' into dpykhtar/new_tokenizer_system
3bf4c23
@dimapihtar
update recipes
e61f0e2 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@dimapihtar
switch to new tokenizers
525de0e 
Signed-off-by: dimapihtar <dpihtar@gmail.com>
@github-actions
Copy link
Contributor

github-actions bot commented Aug 27, 2025

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

@github-actions github-actions bot added the Stale label Aug 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2025

This PR was closed because it has been inactive for 7 days since being marked as stale.

@github-actions github-actions bot closed this Sep 4, 2025
@ko3n1g ko3n1g deleted the dpykhtar/new_tokenizer_system branch February 20, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@dimapihtar