I believe the concept of multimodal token embeddings is quite fitting for my own thinking. Certainly, my 'embeddings' are not always fully formed words, but sometimes they are.