Agree.. There's also some things that you can't do with "bag-of-words" methods, such as handling multi-words because for those, the order of words is relevant.
bag of n-grams can help here, but yes you will still run into a wall quick if you require phrase understanding or context.
That has its own set of problems, namely data sparsity and resulting in high dimensionality vectors, although like you say does give you a little bit of context. I'm not sure that bag of anything is a good solution.