Hacker News Reader

Comment by minimaxir | Hacker News Reader

Google's docs on the APIs are up: https://cloud.google.com/vertex-ai/docs/generative-ai/learn/...

The pricing is also now listed but free during the trial period, although it's annoyingly priced by character: https://cloud.google.com/vertex-ai/pricing#generative_ai_mod...

Assuming ChatGPT's tokens are the equivalent of 4 characters on average (a fair assumption), the pricing of PaLM's chat and embedding APIs are the same cost as OpenAI's equivalents.

View on HN

Why would that be annoying? It’s much easier to understand, predict and truncate appropriately than having to explain all of these different tokenization schemes to devs.

Yeah, everybody agrees on what a character is, right? It's just {an ASCII byte|a UTF8 code unit|a UTF16 code unit|a Unicode code point|a Unicode grapheme}.

I’m not saying it’s easy but it’s much better than tokens IMO. I think bytes would be understandable too.

Bytes are understandable but make no sense from a business point of view. If you submit the same simple query with UTF-8 and UTF-32, the latter will cost 4x as much.

No API accepts input in UTF-32. Nobody uses this on the internet.

At least there are standards for characters. Nothing like that for tokens.

And we think tokens solve that problem? Spoiler alert: they don’t

https://www.reddit.com/r/OpenAI/comments/124v2oi/hindi_8_tim...

They don't but Google could have been more precise with which of the definitions listed by GP they mean by "character".

Per token might be 4 characters on average, but that can vary wildly. Pricing per character is easier to understand and means more flexibility to change tokenisation without affecting pricing. So far OpenAI has charged very different prices per model, but I expect we’ll see more granular changes in the future that might not change pricing… except for changing the tokenisation.

It's happy pricing as a Japanese user, maybe also for Korean and Chinese.

There is a limit of maxOutputTokens, 1024! Is this the true capability of PaLM 2?

However I couldn't find anything about the context length of their model anywhere. And the API didn't tell me how long the prompt could be.

No. Autoregressive models don't have model specific limit to output tokens, it's just when to stop looping.

I guess I only know transformers and how BERT or GPT works, as there would be a limit in the context length. With GPT, you can certainly generate infinite amount of tokens but the previous tokens outside of the maximum context length would be outside of the context window. LLaMa has 2k, GPT-4 has 32k.

Are you saying I can give unlimited tokens to PaLM and generate unlimited amount of tokens? So PaLM doesn't have a context limit?

No, I am not saying that. Since PaLM 2 is a transformer model (they didn't disclose almost anything about the model architecture, but they did disclose that), it has a context length limit. What I am saying is that you can't infer that limit from the limit of maxOutputTokens parameter in the API.

Reply on news.ycombinator.com