I want to be able to view exactly how my input is translated into tokens, as well as the embeddings for the tokens.
"how a piece of text might be tokenized by a language model"
"It's important to note that the exact tokenization process varies between models."
It's the might condition in the description that makes me think the results might not be the exact same as what's used in the live models.