Playground

Tokenizer visualizer

Type a sentence to see how the repository's word-level tokenizer concept splits text into tokens and maps them to vocabulary IDs. This is an illustrative, in-browser demo of the tokenization step — not the trained model itself.