HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by dgellow | Hacker News Reader
Full thread
dgellow
·
Could someone clarify what exactly are those tokens ?
View on HN
winchester6788
·
Length of given text after encoding it with their BPE. In general, you can expect it to be 1.2x-2x the len(words) in your text.
minimaxir
·
The Pricing FAQ notes that it's, on average, 4 characters, which IMO is fair.
Super common punctuation is its own token, however.
Reply on news.ycombinator.com