Hacker News Reader

Comment by thorum | Hacker News Reader

If the rumors about the upcoming Strawberry and Orion models from OpenAI are true - supposedly capable of deep research, reasoning and math - they probably don’t have much to worry about. Not to mention they still have the only fully multimodal model.

View on HN

According to that recent The Information report, Orion is supposed to be just a regular LLM except trained with synthetic data generated via Strawberry. Anthropic et al. have also been working on ways to generate synthetic data (as seen in the success of Sonnet 3.5) so I don't really know if that's going to be a big lead.

And of course the ever-hyped strawberry is supposed to be some sort of tree-of-thought type thing I think, or maybe it's related to https://arxiv.org/abs/2203.14465. Either way, nothing so far has come out that it's a completely novel training technique or architecture, just a gpt-4 scale model with different post-processing.

I'm not sure why people act so mystified about Q*, the name gives it away, it's an obvious reference to A*, the only question is what the nodes in the graph are and what they're using as a Heuristic function.

It's not. It's Quiet STaR.

It is in reference to this:

https://arxiv.org/abs/2203.14465

https://arxiv.org/abs/2403.09629

Q* is also a term from reinforcement learning.

"just a regular LLM except [trained on very different data]."

I'm not saying there's some big moat, anyone can read https://arxiv.org/pdf/2305.20050, but not all synthetic data is created equal. Strawberry I'm sure generates beautiful, valid chain-of-thought reasoning data. Wouldn't surprise me if OpenAI is just significantly ahead of the competition.

Models are only state of the art for 12 to 18 months. There’s not a model in existence today that will have any value in 5 years. They will all be obsolete.

Thus far no one has any moat on their model.

Not on the model but most companies don’t sell models, they sell products and that use models. GPT-3 became obsolete at some point but ChatGPT hasn’t, and has some level of moat with their ecosystem. And cloud providers have moats via lock in, just for blanket products instead of any one model.

What is the only fully multimodal model?

The GPT-4o checkpoint available to the public can see images but not generate them (it can generate prompts for Dalle 3 to use). OpenAI has an internal model with this capability, but if you don't make it a actual product, it doesn't really matter.

It's even more modal than that, 4o accepts text, image, audio, and video input, and produces text, image, or audio output. Video input isn't available yet and was only briefly demoed. Image and video output haven't been demonstrated publicly at all yet.

Rapid productization isn't the priority of most ML devs.

OpenAI has a few examples of image output being produced directly by the GPT-4o checkpoint they have.

>Rapid productization isn't the priority of most ML devs.

It depends if they need money or not, Google has not publish a single image generator that is not a demo.

Reply on news.ycombinator.com