The context comes from the attention mechanism, not from word embeddings.