I've always wondered how stop tokens fit in here. Does the LLM generate a probability for "stop" in addition to every other token in the space? Or is stopping handled heuristically by the outer loop that generates the output tokens sequentially?
The API docs talk about letting you specify your own stop token (like "<!-->") but I don't think "token" is meant in the same sense here.