ParentFull threadkwrobel·I agree. Perplexities (probability of a text) can be compared using different tokenization after normalization.View on HN