Hacker News Reader

Comment by potatoman22 | Hacker News Reader

Google has been doing research into mixture of experts for scaling LLMs. Their GLaM model published in 2022 has 1.7 trillion parameters and 64 experts.

https://icml.cc/media/icml-2022/Slides/17378.pdf

View on HN

Google is jokingly behind in terms of LLMs. They've done a pretty good job at incorporating vision and audio ML models into their ecosystem, but they underestimated language.

How do you know? do you have insider knowledge of this or is it just based on what they share publically?

From what I see in that GPT 3 and 4 was a bit of a rugpull for the industry, now we're all laughing at Google because seemingly they had their hands on the rug for nearly a decade and did nothing - but from the other perspective, maybe they saw the future openai has now brought us and decided against being the pioneers

Amazingly enough, I think this is a bit of it. Some powerful enough people at Google became concerned about implications, including around "hallucinations", "poisoning", etc., and decided to put this sort of research on something of a backburner - justified, in part, by a lack of some obvious easy interfacing of this with search (scaling, hallucinations, etc.).

Of course, the 'wonderful' thing about humans / "independent agents with survival drives in competitive game-theoretic type scenarios" is: if enough people / "agents" have access / opportunity, someone WILL "push the button".

It's just delicious ... the same kinds of patterns over and over - "oh, we should really do something about X / nobody should have power like X, ... but, there's no stopping it, ... oh well".

And, the "rules" really are subtly, many levels down, in place, to make it apparently impossible to not get trapped, one way or another.

(Anyway... [Cartman voice] Screw you guys, I'm going to my other planet...)

as a fun ancedote, the Google Bard's implicit code execution update from *last month*, advertised by Sundar... no longer works https://twitter.com/swyx/status/1678495067663925248

i'd love to know whats going on in that team.

Probably safety-driven terror. They really really want to get their bots going, but in every single meeting some PM or other concerned engineer talks about safety and f**s up the entire meeting.

They even made the bot not respond to arithmetics questions because the bot is bad at this, lol. Someone who knows how to modify the bot had actually spent their time on something as unimportant as that.

Bard being bad at anything else doesn't seem to stop it. It hallucinates at the drop of the hat. Asking it almost any question implying X nonexistent thing exists causes it to make that thing up.

> i'd love to know whats going on in that team.

Seems like a good time to rewatch Silicon Valley and watch Hooli scramble to keep up.

quite, some aspects of emergent behaviour are exciting, others must be a painful learning experience.

Their translation service is based on llms and is commercially a successful product.

Transformer models rather than LLMs surely. ChatGPT behaves nothing like Google Translate.

The T in ChatGPT stands for Transformers. The similarity between the OG Transformer from 2017 and GPT3 (and other modern LLMs) is pretty big

The data, size and training process are what's different.

The point is LLMs are built with Transformer architecture. They're all transformers, attention is an integral part of building worthwhile, contextual answers.

Yeah, but LLMs are more specific, so it's a subset.

Also, LLMs use "in-context learning" ie you actually ask it "hey, translate this" and then it has a conversation with you where you can ask to clarify or provide word definitions.

Google Translate is more hardcoded; a big problem with it is that it can't explain any of the decisions it's made or show its uncertainty about anything.

Of course, neither of these are reliable since they hallucinate.

I favor ChatGPT over Google Translate now. ChatGPT has the added benefit it is focused on providing helpful answers, so if it comes across something almost untranslatable, it will be able to address that and provide an explanation in the style of a footpage note.

Maybe, but they're relatively open with their research, which is great. They also made BERT and released it for free.

Reply on news.ycombinator.com