Gemini "duck" demo was not done in realtime or with voice
twitter.com
twitter.com
My group (3 of us) bought a moisture sensor to plug into the pi, and had the idea to make a "flood detection system" that would be housed under a bridge, and would send an email to relevant people when the bridge home from work is about to flood.
So for our demonstration, we had a guy in the back of the class with gmail open ready to send an email saying some variation of "flood warning". Our script was literally just printing lines with wait statements in between. Running the script, it prints to the screen "awaiting moisture", and after 3 seconds it will print "moisture detected". In that 3 seconds I dip the sensor into the glass of water. Then the script would wait a few more seconds before printing "sending email to xxx@yyy.com". We then opened up our email, our mate at the back of the room hit send, and an email appeared saying flood warning, and we would get full marks.
We’d set up a dummy HMI and have someone pressing buttons on it for the demo, and someone in the next room manually driving outputs and inputs to make it seem like it was working. Very common.
Garbage in, garbage out.
Looking at this a different way, they gave first-year students, likely with no established pre-requistites, an open-ended project with fixed hardware but no expectation to submit the final project for review. If they wanted to verify the students actually developed a working program, they could have easily asked for the Pi's to be returned along with the source code.
A project like this was likely intended to get the students to think about the "what" and not worry so much about the "how." Faking it entirely may have gone a bit further than intended, but would still meet the goal of getting the students to think about what they could do with this computer (if they knew how)
While university instructors can vastly underestimate student's creativity, they are, generally speaking, not stupid. At the very least, they know if you don't tell students to submit their work, you can often count on them doing as little as possible.
Literally all that matters is that they passed.
The Wiimote connection was the star of the show by a long shot :P
The amount of fumbles is monumental.
2008 is late to the party for a docs competitor! Microsoft got the runaround by Google and after Google launched docs they could have clobbered Microsoft which kind of failed to respond properly in kind, but they didn’t push the platform hard enough to eat the corporate market share, and didn’t follow up with a share point alternative that would appeal to the enterprise, and kind of blew the opportunity imo.
I mean to this day Google docs is free but it still hasn’t unseated Word in the marketplace, but the real killer app that keeps office on top is Excel, which some companies built their entire tooling around.
It’s crazy interesting to look back and realize how many twists there were leading us to where we are today.
Btw it was Office Server or Sharepoint Portal earlier (this is like Frontpage days so like 2001?) and Microsoft called it Tahoe internally. I don’t think it became Sharepoint until Office 365 launched.
The XMLHTTP object launched in 2001 and was part of the dhtml wave. That gave a LOT of the capabilities to browsers that we currently see as browser-based word processing, but there were efforts with proprietary extensions going back from there they just didn’t get broad support or become standards. I saw some crazy stuff at SGI in the late 90s when I was working on their visual workstation series launch.
https://www.zdnet.com/article/netdocs-microsofts-net-poster-...
So why did they never release that and went with Office 365 instead?
A product requires commitment, it requires grind. That 10% is the most critical one, and Google persistently refuses to push products across the finish line, just giving up on them and adding to the infamous Google Product Graveyard.
Honestly, what is the point? They could just maintain the core search/ads and not pay billions of dollars for tens of thousands of expensive engineers who have to go through a bullshit interview process and achieve nothing.
tech was based on an acquired company, Google just abused their search monopoly to make it more popular(same thing they did with YT). This has been the strategy for every service they've ever made, Google really hasn't launched a decent in-house product since Gmail and even that was grown using their search monopoly as free advertising
>Google Docs originated from Writely, a web-based word processor created by the software company Upstartle and launched in August 2005
What about Chrome? And Chromebooks?
This is what Google has always cared about. Bring application to the billions of users.
People are forgetting Google is the most profitable AI company in the world right now. All of their products use ML and AI.
So who is losing?
The goal of Gemini isn't to build a chatbot like ChatGPT despite Google having Bard.
The goal for Gemini is to integrate it into those 10 products they have with a billion users.
> So who is losing?
The people who use their products, which are worse than they’ve been in decades? The people who make the content Google now displays without attribution on search results?
Is that supposed to be a vote of confidence for the current state of Google search?
Shopped it around VCs. Got laughed out of all the meetings. "Companies storing their documents on the Internet?! You're out of your mind!"
Globe dot com was basically Facebook, but the critical mass wasn't there. Nor were the smartphones.
And this business is so totally different to Google in every way imaginable.
Senior Managers love customer support, SLAs - Google loves automation. Two worlds collide.
Word and Excel have been dominant since the early 1980s. Google has never had a real shot in the space.
Google has the problem that it's typically the first to encounter a problem, and it has the resources to approach it (from search), but the incentive to monetize it (to get away from depending entirely on search revenue). And, management.
They can't do anything that threatens their main income. They are tied to ads and ads technology, and can't do anything about it.
Microsoft had a crisis and that drives focus. Google... they probably mistreat their good employees if they don't work on ads.
AI is a threat there, but it'd require an AI company to transform the culture of Internet use to stop people 'Googling', and that will require two things: something significantly better than Google Search that's worth switching to, and a company that is willing to reject whatever offer Google makes to buy it. Neither is very likely.
Nadella is an all time great CEO. Pichai is an uninspired MBA-type.
Of all the things, this.
I use both Google and Microsoft office products. One thing that strikes you is just how feature rich Microsoft products are.
Google doesn't look like is serious about making money.
I squarely blame rockstar product managers and OKRs for this. Not everything can be a 1000% profitable product built in the next quarter. A lot of things require small continuous improvement and care over years.
I agree with your general post but I disagree with this. Tesla's FSD is so far behind Google it's almost negligent on the part of Tesla despite having so much more data.
Besides collaborative reuniting is a no feature and there is much more important stuff than this for a word processor to be useful.
Well, that is trully shocking.
Google Keynote feels like it's emulating the Apple keynote from 5 years ago.
And the Apple keynote looks like robots just out of an uncanny valley pretending to be humans - just like keynotes might look in 5 years, but actually made by AI. Apple is always ahead of the curve in keynote trends.
That's what Apple keynotes feel like now. It seems like each year, they're trying to make their presentations even more essentially 'Apple.' They crossed the uncanny valley a long time ago.
It’s the stillness between “beats” that does it, I think, and the very-constrained and repetitive motion.
What does that mean and why is it bad?
Diversity in marketing is used because, well, your desired market is diverse.
I don't know what it means for it to be surgically precise, though.
I don't mean to imply that companies should avoid displays of diversity, I just mean that it's obvious when it's inauthentic. Virtue signaling in exchange for business is not progress.
Now, I don't know how diverse the AI workforce is at Google, but the YT thumbnails show precisely 50% of white men. Maybe that's what the parent meant by "surgically precise".
They oversold its capabilities, but it does still seem that multi-modal models are going to be a requirement for AI to converge on a consistent idea of what kinds of phenomena are truly likely to be observed across modalities. So it's a good step forward. Now if they can just show us convincingly that a given architecture is actually modeling causality.
"do you believe that a pocket of hot air would lead to lower air pressure causing my plane to stall?"
he could barely even phrase the question correctly because it was so awkward. just embarrassing.
I believe it is Hinton that prefers “confabulation” to “hallucination” because it’s more accurate. The example in the discussion about hallucination/confabulation was that of someone who had been present in the room during Nixon’s Watergate conversations. Interviewed about what he heard, he provided a narrative that got many facts wrong (who said what, and what exactly was said). Later, when audio tapes surfaced, the inaccuracies in his testimony became known. However, he had “confabulated truthfully”. That is, he had made up a narrative that fit his recall as best as he was able, and the gist of it was true.
Without the ability to confabulate, he would have been unable to tell his story.
(Incidentally, because I did not check the facts of what I just recounted, I just did the same thing…)
It simply spits out whatever output sequence it feels is most likely to occur after your input sequence. How it defines “most likely” is the subject of much research, but to optimize for factual correctness is a completely different endeavor. In certain cases (like coding problems) it can sound smart enough because for certain prompts, the approximate consensus of all available text on the internet is pretty much true and is unpolluted by garbage content from laypeople. It is also good at generating generic fluffy “content” although the value of this feature escapes me.
In the end the quality of the information it will get back to you is no better than the quality of a thorough google search.. it will just get you a more concise and well-formatted answer faster.
I think you're slightly mischaracterising things here. It has potential to be at least slightly and possibly much better than that. This is evidenced by the fact it is much better than chance at answering "novel" questions that don't have a direct source in the training data. Why it can do it is because at a certain point, to solve the optimisation problem of "what word comes next" the least complex strategy actually becomes to start modeling principles of logic and facts connecting them. It is not in any systematic or reliable way so you can't ever guarantee when or how well it is going to apply these, but it is absolutely learning higher order patterns than simple text / pattern matching, and it is absolutely able to generalise these across topics.
I would say it’s worse than Google search. Google tells you when it can’t find what you are looking for. LLMs “guess” a bullshit answer.
What if the input sequence says "the following is truth:", assuming it skillfully predicts following text, it would mean telling the most likely truth according to its training data.
This was all fake. You are taking a collection of cherry picked prompt engineered examples, then dramatizing them for maximum shareholder hype. The music example was just outputting a description of a song, not the generated music we heard in the video.
It’s one thing to release a hype video with what-ifs and quite another to claim that your new multi-modal model is king of the hill then game all the benchmarks and fake all the demos.
Google seems to be in an evil phase. OpenAI and MS must be quite pleased with themselves.
1) Forward looking demoes that demonstrate the future of your product, where it’s clear that you’re not there yet but working in that direction
or
2) Demoes that show off current capabilities, but are scripted and edited to do so in the best light possible.
Both of those are standard practice and acceptable. What Google did was just wrong. They deserve to face backlash for this.
My friend, all these large corporations are going to get away with exactly as much as they can, for as long as they can. You're implying there's nothing to do but wait until they grace us with a "not evil phase", when in reality we need to be working on restoring our anti-monopoly regulation that was systematically torn down over the last 30 years.
Given the massive data volume in videos, I assumed it processed video into pictures by extracting a frame per second or something along those lines, while still taking the entire video as the initial input.
Turns out, it wasn't even doing that!
Doing all these hype videos just for the sake of satisfying shareholders or whatever is just making me loose trust in their research division. I don't think they did anything like this when they released Bert.
If I demoed swype texting as it functions in my day to day life to someone used to a querty keyboard they would never adopt it
The rate at which it makes wrong assumptions about the word, or I have to fix it is probably 10% to 20% of the time
However because it’s so easy to fix this is not an issue and it doesn’t slow me down at all. So within the context of the different types of text Systems out there, I t’s the best thing going for me personally, but it takes some time to learn how to use it.
This is every product.
If you demonstrated to people how something will actually work after 100 hours of habituation and compensation for edge cases, nobody would ever adopt anything.
I’m not sure how to solve this because both are bad.
(Edit: I’m keeping all my typos as meta-comment on this given that I’m posting via swype on my phone :))
Unfortunately iOS text editing is also completely worthless. It forces strange selections and inserts edited text in awkward ways.
I’m a QWERTY texter but text entry on iOS is a complete disaster that has only gotten worse over time.
But that's a different issue than LLM hallucinations.
With Swype, you already know what the correct output looks like. If the output doesn't match what you wanted, you immediately understand and fix it.
When you ask an LLM a question, you don't necessarily know the right answer. If the output looks confident enough, people take it as the truth. Outside of experimenting and testing, people aren't using LLMs to ask questions for which they already know the correct answer.
It is the main reason that handwriting recognition did not displace keyboards. Once the handwriting is converted to text, it’s easier to fix errors with a pointer and keyboard. So after a few rounds of this most people start thinking: might as well just start with the pointer and keyboard and save some time.
So the question is, how easy is it to detect and correct errors in generative AI output? And the unfortunate answer is that unless you already know the answer you’re asking for, it can be very difficult to pick out the errors.
This is the sort of demo that 1) gives people a misleading idea of what the product can actually do; and 2) ultimately contributes to the inevitable cynical backlash.
If the product is really great, people can see it in a realistic demo of its capabilities.
Maybe you can spice up a demo, but misleading to the point of implying things are generated when they're not (like the audio example) is pretty bad.
Except actual good ones, like ChatGPT or Gmail (by their time).
In your Swype analogy, it would be as if Swype works by having to write out on a piece of paper the general goal of what you're trying to convey, then having to write each individual letter on a Post-it, only for you to then organize these Post-its in the correct order yourself.
This process would then be translated into a slick promo video of someone swiping away on their keyboard.
This is not a matter of “eh, it doesn't 100% work as smooth as advertised.”
0: https://techcrunch.com/2023/12/07/googles-best-gemini-demo-w...
[1] https://www.bloomberg.com/opinion/articles/2023-12-07/google...
[2] https://www.bloomberg.com/opinion/articles/2023-12-07/google...
I am curious, if the voice, email, chat, and shortly video can all be entirely generated in real or near real time, how can we be sure that remote employee is actually not a full or partially generated entity?
Shared secrets are great when verifying but when the bodies are fully remote - what is the solution?
I am traveling at the moment. How can my family validate that it is ME claiming lost luggage and requesting a Venmo request?
PGP
(I say this in jest, as a PGP user)
Hopefully realtime interaction will be part of an app soon. Doesn’t seem like there would be too many technical hurdles there.
They do disclose most of the details elsewhere, but the video itself is produced and edited in such a way that it's extremely misleading. They really want you to think that it's responding in complex ways to simple voice prompts and a video feed, and it's just not.
That's different from "Gemini was shown selected still images and not video".
The video fooled many people, including myself. This was not your typical super optimized and scripted demo.
This was blatant false advertising. Showing capabilities that do not exist. It’s shameful behavior from Google, to be perfectly honest.
In my estimation, given the context around AI-generated content and general fakery, this video was deceptive. The only impressive thing about the video (to me) was how snappy and fluid it seemed to be, presumably processing video in real time. None of that was real. It's borderline fraudulent.
The video has a disclaimer that it was edited for latency.
And good speech-to-text and text-to-speech already exists, so building that part is trivial. There's no deception.
So then it seems like somebody is pressing a button to submit stills from a video feed, rather than live video. It's still just as useful.
My main question then is about the cup game, because that absolutely requires video. Does that mean the model takes short video inputs as well? I'm assuming so, and that it generates audio outputs for the music sections as well. If those things are not real, then I think there's a problem here. The Bloomberg article doesn't mention those, though.
> The video has a disclaimer that it was edited for latency.
There was no disclaimer that the prompts were different from what's shown.
> And good speech-to-text and text-to-speech already exists, so building that part is trivial. There's no deception.
Look at how many people thought it can react to voice in real-time - the net result is that a lot of people (maybe most?) were deceived. And the text prompts were actually longer and more specific than what was said in the video!
> somebody is pressing a button to submit stills from a video feed, rather than live video.
Somebody hand-picked images to convey exactly the right amount of information to Gemini.
> Does that mean the model takes short video inputs as well? I'm assuming so
It was given a hand-picked series of still images with the hands still on the cups so that it was easier to understand what cup moved where.
Source for the above: https://developers.googleblog.com/2023/12/how-its-made-gemin...
But the most impressive part of the demo, was the way the LLM just seemed to know when to jump in with a response. It appeared to be able to wait until the user had finished the drawing, or even jumping in slightly before the drawing finished. At one point the LLM was halfway though a response and then saw the user was now colouring the duck in blue, and started talking about how the duck appearing to be blue.
The LLM also appeared to know when a response wasn't needed because the user was just agreeing with the LLM.
I'm not sure how many people noticed that on a conscious level, but I positive everyone noticed it subconsciously, and felt the interaction was much more natural.
As you said, good speed-to-text and speech-to-text has already been done, along with multi-model image/video/audio LLMs and image/music generation. The only novel thing google appeared to be demonstrating and what was most impressive was this apparent natural interaction. But that part was all fake.
> My main question then is about the cup game, because that absolutely requires video.
They did it with carefully timed images, and provided a few examples first.
> I'm assuming so, and that it generates audio outputs for the music sections as well
No, it was given the ability to search for music and so it was just generating search terms.
Here's more details:
https://developers.googleblog.com/2023/12/how-its-made-gemin...
...OpenAI solved this by generating LLM text for you to wade through?
Though because you don’t see the answers it doesn’t show you, it’s hard to really validate the quality, so I’m still wary, but when I look for specific stuff it tends to find it.
Turns out it was all staged.
Lost a lot of trust after that.
Google is stuck in innovators dilemma.
They make 300B of revenue which ~90% is ads revenue.
Their actual mission that management chain optimizes for is their $ growth.
A superior AI model that gives the user exactly what they want would crash their market cap.
Microsoft has tons of products with Billion+ profit, Google has only a handful and other than cloud they all tie to Ads.
Google is addicted to ads. If chrome adds a feature that decreases ad revenue, that team gets a stick.
Nothing at Google should jeopardize their ad revenue.
AI is directly a threat to Google’s core business model - ads. It’s obvious they’re gonna half ass it.
For OpenAI, AI is existential for them. If they don’t deliver, they’ll die.
It's the single biggest problem with LLMs and Gemini isn't solving it. You simply can't rely on them when correctness is important. Even when the model has the knowledge it would need to answer correctly, as in this case, it will still lie.
The false statement is after it says the duck floats, it continues "It is made of a material that is less dense than water." This is false; "rubber" ducks are made of vinyl polymers which are more dense than water. It floats because the hollow shape contains air, of course.
Search or even asking other expert human beings are prone to provide incorrect results. I'm unsure where this expectation of 100% absolute correctness comes from. I'm sure there are use cases, but I assume it's the vast minority and most can tolerate larger than expected inaccuracies.
As an example, when most people say a balloon's lighter than air, they mean an inflated balloon with hot air or helium, but you catch their meaning and don't rush to correct them.
I suppose there’s two possible solutions: one is a new training or inference architecture that somehow understand “facts”. I’m not an expert so I’m not sure how that would work, but from what I understand about how a model generates text, “truth” can’t really be a element in the training or inference that affects the output.
the second would be a technology built on top of the inference to check correctness, some sort of complex RAG. Again not sure how that would work in a real world way.
I say it might be fundamental to how the model works because as someone pointed out below, the meaning of the word “material” could be interpreted as the air inside the duck. The model’s answer was correct in a human sort of way, or to be more specific in a way that is consistent with how a model actually produces an answer- it outputs in the context of the input. If you asked it if PVC is heavier than water it would answer correctly.
Because language itself is inherently ambiguous and the model doesn’t actually understand anything about the world, it might turn out that there’s no universal way for a model to know what’s true or not.
I could also see a version of a model that is “locked down” but can verify the correctness of its statements, but in a way that limits its capabilities.
Preserving the original comment so the replies make sense:
---
I think it's a stretch to say that's false.
In a conversational human context, saying it's made of rubber implies it's a rubber shell with air inside.
It floats because it's rubber [with air] as opposed to being a ceramic figurine or painted metal.
I can imagine most non-physicist humans saying it floats because it's rubber.
By analogy, we talk about houses being "made of wood" when everybody knows they're made of plenty of other materials too. But the context is instead of brick or stone or concrete. It's not false to say a house is made of wood.
LLMs right now are most practical for generating templated text and images, which when paired with an experienced worker, can make them orders of magnitude more productive.
Oh, DALL-E created graphic images with a person with 6 fingers? How long would it have taken a pro graphic artist to come up with all the same detail but with perfect fingers? Nothing there they couldn't fix in a few minutes and then SHIP.
- Just after that it doesn't translate the "rubber" part
- It states there's no land nearby for it to rest or find food in the middle of the ocean: if it's a rubber duck it doesn't need to rest nor feed. (That's a missed opportunity to mention the infamous "Friendly Floatees spill"[1] in 1992 as some rubber ducks floated to that map position). Although it seems to recognize geographical features of the map, it fails to mention Easter Island is relatively nearby. And if it were recognized as a simple duck — which it described as a bird swimming in the water — it seems oblivious to the fact that the duck might feed itself in the water. It doesn't mention either that the size of the duck seems abnormally big in that map context.
- The concept of friends and foes doesn't apply to a rubber duck either. Btw labeling the duck picture as a friend and the bear picture as a foe seems arbitrary (e.g. a real duck can be very aggressive even with other ducks.)
Among other things, the astronomical riddle seems also flawed to me: it answered "The correct order is Sun, Earth, Saturn".
I'd like for it to state :
- the premises it used, like "Assuming it depicts the Sun, Saturn and the Earth" (there are other stars, other ringed-planets, and the Earth similarity seems debatable)
- the sorting criteria it used (e.g. using another sorting key like the average distance from us "Earth, Sun, Saturn" can be a correct order)
Am I missing something?
I asked both Bard (Gemini at this point I think?) and GPT-4 why ducks float, and they both seemed accurate: they talked about the density of the material plus the increased buoyancy from air pockets and went into depth on the principles behind buoyancy. When pressed they went into the fact that "rubber"'s density varies by the process and what it was adulterated with, and if it was foamed.
I think this was a matter of the video being a brief summary rather than a falsehood. But please do point out if I'm wrong on the rubber bit, I'm genuinely interested.
I agree that hallucinations are the biggest problems with LLMs, I'm just seeing them get less commonplace and clumsy. Though, to your point, that can make them harder to detect!
Maybe AI correctness will be similar to automobile safety. It didn’t take long for both to be recognized as fundamental issues with new transformative technologies.
In both cases there seems to be no silver bullet. Mitigations and precautions will continue to evolve, with varying degrees of effectiveness. Public opinion and legislation will play some role.
Tragically accidents will happen and there will be a cost to pay, which so far has been much higher and more grave for transportation.
It certainly isn't how I would phrase it, and I wouldn't count air as what something is made of, but...
Soda pop is chocked full of air, it's part of it! And I'd say carbon dioxide is a part of the recipe, of pop.
So it's a confusing world for a young LLM.
(I realise it may have referenced rubber prior, but it may have meant air... again, Devil's advocate)
The models are too useful to say, "don't use them at all." Hopefully people will heed the warnings of how they can hallucinate, but further than that I'm not sure what more you can expect.
If you go to such technical routes your definition is wrong too. It doesn't float because it contains air. If you poke in the head of the duck it will sink. Even though at all times it contains air.
I loved it when the lawyers got busted for using a hallucinating LLM to write their briefs.
If I pushed the model further on the composition of a rubber duck, and it failed to mention its construction, then it’d be lying.
However there is this disgusting part of language where a statement can be misleading, technically true, not the whole truth, missing caveats etc.
Very challenging problem. Obviously Google decided to mislead the audience and basically cover up the shortcomings. Terrible behaviour.
While I agree, I wouldn't call proofs or concepts and demos pointless. They often illustrate a goal or target functionality you're working towards. In some cases it's really just a matter of allotting some time and resources to go from a concept to a product, no real engineering is needed, it all exists, but there's capital needed to get there.
Meanwhile some proof of concepts skip steps and show higher level function that needs some serious breakthrough work to get to, maybe multiple steps of that. Even this is useful because it illustrates a vision that may be possible so people can understand and internalize things you're trying to do or the real potential impact of something. That wasn't done here, it was embedded in a side note. That information needs to be before the demo to some degree without throwing a wet blanket on everything and needs to be in the same medium as the demo itself so it's very clear what you're seeing.
I have no problem with any of that. I have a lot of problems when people don't make it explicitly clear beforehand that it's a demo and explain earnestly what's needed. Is it really something that exists today in working systems someone just needs to invest money and wire it up without new research needed? Or is it missing some breakthroughs, how many/what are they, how long have these things been pursued, how many people are working on them... what does recent progress look like and so on (in a nice summarized fashion).
Any demo/poc should come up front with an earnest general feasibility assessment. When a breakthrough or two are needed then that should skyrocket. If it's just a lot of expensive engineering then that's also a challenge but tractable.
I've given a lot of scientific tech demonstrations over the years and the businesses behind me obviously want me to be as vague as possible to pull money in. I of course have some of those same incentives (I need to eat and pay my mortgage like everyone else). None-the-less the draw of science to me has always been pulling the veil from deception and mystery and I'm a firm believer in being as upfront as possible. If you don't lead with disclaimers, imaginations run wild into what can be done today. Adding disclaimers helps imaginations run wild about what can be done tomorrow, which I think is great.
Ultra is not yet available.
"There's no such thing as bad publicity."
Literally everyone I know who works at Google hates their job and are completely checked out.
If it had been real, that is.
Implying they’ve solved single token latency, however, is very distasteful.
Very misleading and sad Google would so obviously fake a demo like this. Mentioning in the description that it's edited is not really in the realm of doing enough to make clear the fakery.
mea cupla i should have looked at the bottom of the description box on youtube where it probably says "this demonstration is based on an actual interaction with an LLM"
I wish people stop posting Twitter messages to HN and provide a link directly to the original article. What's next, post on HN on an Instagram post?
And another: https://techcrunch.com/2023/12/07/googles-best-gemini-demo-w...
edit: s/stuck/stock
[Tech CEO read instructions to "show empathy" from his assistant via Slack]
CEO: I hear you. It must be very hard for you.
Shooter: Of course you fucking hear me, we're on the phone! Talk like a normal person!
But then I soon noticed some things that were too smooth, so seemed at best to be cherry-picked interactions occasionally leaning on hand-crafted situation handlers. Or, it turns out, faked.
Regardless of disclaimers, this video seems misleading to be releasing right now, in the context of OpenAI eating Google's lunch.
Everyone is expecting Google to try to show they can do better. This isn't that. This isn't even an mocked-up interaction future of HCI concept video, because it's not showing a vision of what people want to do --- it's only showing a demo of technical capabilities.
It's saying "This is what a contrived tech demo (not application vision concept) could look like, but we can't do it yet, so we faked it. Hopefully, the viewer will get the message that we're competitive with OpenAI."
(This fake demo could just be an isolated oops of a small group, not representative of Google's ability to rise to the current disruption challenge, I don't know.)
BUT
The fact this wasn’t realtime or with voice is not the issue. Voice to text could absolutely capture this conversation easily. And from what I’ve seen Gemini seems quicker than GPT4
Being able to respond quicker and via voice chat is not actually a big deal.
The underlying performance of the model is what we should be focussing on
One of the major issues in LLMs is the economics; a lot of people suspect ChatGPT loses money on every user, or at least every heavy user, because they've got a big model and A100 GPUs are expensive and in short supply.
They're kinda reluctant to have customers, with API rate limits galore, and I've heard people claiming ChatGPT has lost the performance crown having switched to a cheaper-to-run model.
If google had a model that operated on video in realtime, that would imply they've got a model that performs well, and is also very fast or that their 'TPUs' outperform the A100 quite a bit, either of which would be a big step forward.
It goes from a model being able to infer a great deal at the level of human intelligence to a model that needs to be fed essential details, and that doesn't do much inferring.
I get the feeling that many here on HN who are just shrugging it off don't realize how much of the “demo” was faked. Here’s a decent article that goes into it some more: https://techcrunch.com/2023/12/07/googles-best-gemini-demo-w...
Corporate tech demo exaggerates actual capabilities and smoothes over rough edges? Impossible, this is unprecedented!!
The Apple vs Google brand war is so tiresome. Let's focus on the tech.
As much as I hate it, this is absolutely fine by our society's standards. https://www.theregister.com/AMP/2014/02/14/apple_prevails_in...
The former is considered Puffery[1] and is completely legal, and the latter is straight up lying.
0: https://techcrunch.com/2023/12/07/googles-best-gemini-demo-w...
The entire launch felt like a concentrated effort to “appear” competitive to OpenAI. Google was splitting hairs talking about low single digit percentage improvement in benchmarks. Against a model that has been out for over 6 months.
I have never been so unimpressed with them. Not only has OpenAI managed to snag this one from under Google’s nose, IMO - they seem to be defending their lead quite well. Now that is something unmistakably remarkable. Color me impressed!
The more these models improve the more we will want less friction and faster interactions, this means that in the long term having to open an app and ask a question is not gonna fly compared to just pointing your phone camera to something, asking a question and getting an answer that's tailored to everything Google knows about you in real time.
Apple will most likely also roll their own in house solution for Siri instead of relying on an external company. This leaves OpenAI and the other small companies not just competing for the best models but also on how to put them in front of people in the first place and how to get access to their personal information.
I think you have too much information to form a reasonable opinion on the situation. Google is using editing techniques and specific scripting to try to demonstrate they have a sufficiently powerful general AI. The magnitude of this claim is huge, and the fact that they're faking it should be a likewise enormous scandal.
To sum this up "well I guess they're doing better than XYZ" discounts the absurd context of all this.
But right now, all the bits needed to do this already exist (just need to be assembled and -to be fair- given a LOT of polish), so it would be somewhat reasonable to think that someone had actually Put In The Work already.
The thing is that for all practical intents and purposes, if I can't try it, it doesn't exist. If they claim it exists, they should show it, a video or a few cherry-picked examples don't prove anything. It's easy to make a demo making even Eliza look like AGI by asking the right questions.
Problem showed in the video was reused in a recent competition (so could have been available in the dataset).
Gemini Ultra is barely beating ChatGPT in their manufactured benchmarks, and this is all that they got.
What this means is that those who are saying, including people at Google, that they have better models but are not releasing them in the name of AI safety, imply that, at least in the realm of LLMs, Google DeepMind had nothing all along.
Shame on them :(
But neither AI not climate are there yet…
All companies are just yelling that they're "in" the AI/LLM game.. If they don't, share prices will drop.
"Absolutely mindblowing. The amount of understanding the model exhibits here is way way beyond anything else."
Can we change the URL to the real article if it still exists?
I wasn't at all under the impression they were showcasing TTS or low latencies as product features. I don't find the marketing misleading at all, and find these criticisms don't hit the mark.
[0] - https://www.bloomberg.com/opinion/articles/2023-12-07/google...
Brilliant idea!
I for one, welcome our new quack overlords.
Still I would distinguish between genuine optimism and cynical exaggeration. If you go back and read what was being said in and around the 70s about technology and watch the demos they gave…it always makes me feel a little sad for them. People really thought so many things were right around the corner, e.g. if a computer can beat a human star chess player, surely understanding natural language or driving a car will be easy. There were so many things being made like primitive robots that could follow a line painted on the floor, that in a controlled environment seemed so promising but the realization of the vision would take decades longer than predicted. Without their efforts we would not be where we are today.
This AI stuff is becoming super cool and the last thing we need is the usual charlatans that surround technology stuff. They should be ashamed of themselves, but I know who called for it - not the engineers, but the "management" layer and they're incapable of feeling shame.
As soon as I saw that opening paragraph from Sundar and how it was written I knew that Gemini is going to be a steaming pile of shit.
They should have watched the GPT-4 announcement from OpenAI again. That demo Greg Brockman did with converting a sketch on a piece of paper to a CodePen from a Discord channel, with all the error correcting and whatnot, is how you launch a product that's appealing to users.
TechCrunch, Twitter and some other sites (including HN i guess) are already piling on to this and by Monday things will go back to how they were and Google will have to go back to the drawing board to figure out another way to relaunch Gemini in the future.
Anyone with half a brain could see through this "demo." It was vastly too uncanny to be real, to the point that it was poorly setup. Google should be ashamed.
So what?
The fact that they did not fact check the videos again makes me not particularly confident in the quality of Google's work. The bit where the model misinterpreted music notation (the circled area does not mean "piano"), and the "less dense than water" rubber duck are beyond the pale. The SVG demo where they generate a South Park looking tree looks like a parody.