HN
Hacker News
Top
New
Best
Ask
Show
Jobs
Comment by weird-eye-issue | Hacker News Reader
Parent
Full thread
weird-eye-issue
·
Yeah... it obviously uses the appropriate tokenizer
View on HN
spencerchubb
·
Some companies like Anthropic haven't publicly released the tokenizer used in their API, thereby making it impossible for this library to use the appropriate tokenizer in all cases. Be careful about how you use the word 'obviously'
Reply on news.ycombinator.com