[Tokenizer] Add batched tokenization for queued requests

[taking-lying-flat]

Summary

• Batch queued prompts into a single tokenizer call.
• Preserve chat-template preprocessing and existing special-token behavior.
• Return individual int32 tensors without padding.
• Add tests for batched prompts, mixed chat/plain inputs, and empty batches.