feat: add skip_tokenizer option for pre-tokenized input#50

ChuanLi1101 wants to merge 1 commit intomainROCm/ATOM:mainfrom

feature/skip-tokenizer-cleanROCm/ATOM:feature/skip-tokenizer-cleanCopy head branch name to clipboard