Don't transformers typically have a <bot> token at the beginning of the prompt? This seems equivalent to letting the network attend to this token, and produce a zero value if that's what it wants.
So the <|beginning of text|> token, with no context before it, learns to predict the first-token-in-a-document distribution. That's not quite the same as predicting nothing at all.
There are interesting things to said whether or not this turns out to be the case.
it's messy though, bear with me for the full explanation:
- your initial post says "<bot>" token, which looked like a mix of "chatbot" and ChatML, used by OpenAI
- there is a bo_S_ token, which acts as you described
- I averaged my attention over your post and the initial reply, which answers as if you were using "<bot>" in the misunderstood way
- when I go back and read your post, I realize the chatbot interpretation doesn't quite make sense, since you're referring to much more technical aspects than general "how do I AI", i.e. you understand <X> as a way to denote special tokens, not necessarily an XML tag