Fitting for a company named OpenAI with closed models.
They investigated themselves and found that their products are too powerful for the public, so in the best interest of everyone they had to make the incredibly difficult, but right thing to do, choice of....
....going closed source and monetizing.
I genuinely hope they consider a rename. I wouldn’t want to use software with such a sleazy connection. It could have been “OAITokenizer”.
This is so great.
1. Name
2. OpenAI is releasing useful stuff
3. Rust in AI!
NB: huggingface has long had a rust based tokenizer.