I think the tokens are simply one hot encoded. Though some sort of word2vec embedding is also situationally plausible