The progress and usefulness of these products is absolutely incredible.
I'd just finished reading The Singularity is Near for the second time too...
I love kurzweil but his estimates of timeline are often pretty over optimistic, so I'd be really wondering.
https://www.metaculus.com/questions/3479/date-weakly-general...
When I read the book the first time about three years ago I thought "2045" is about right.
When I saw DALL-E 2 I thought "2030".
When I saw GPT4 I thought "2026".
- Code interpreter, function calling were already possible on any sufficiently advanced LLM that could follow instructions well enough to output tokens in a rigidly parseable format, which could then be fed into a parser, and its output fed back to the LLM. It was clunky to do with online APIs like ChatGPT, but still eminently possible.
- Custom chatbots were easy to build before, and services to build them (like Poe.com) existed before.
- Likewise outputting JSON just requires a good instruction following AI, that can output token probabilities, along with a schema validator that always picks a token that results in schema-conforming JSON
- GPT4-128k seems to be revolutionary, but Claude-100k already existed, and considering LLM evaluation is quadratic wrt context size, they are probably using some tricks to extend the context, they are not 'full' tokens (I'd be happy to be proven wrong). While having a huge context is useful, for coding, a 8k context can be enough with some elbow grease (like filling the context with a 2-3 deep recursive 'Go To Definition' for a given symbol), so that the AI receives the right context.
- Dall-E 3 seems to be the most revolutionary, but after playing with it, it has much improved compositional ability over SD, but it's still prone to breakdowns
Overall I feel like todays announcements were polish and refinement over last year's bombshell breakthroughs.
* GPT-4-128k. Sure, Claude exists, but it's closer to GPT-3.5 than 4 IMO. TBD how well 128k context works given that classic attention scales quadratically (so they're presumably using something else) but given how good OpenAI's models tend to be I'm willing to give them the benefit of the doubt.
* Pushed GPT-3.5 finetuning to 16k context (up from 4k when it was released this summer). IME 3.5 finetunes are very useful, very fast, very cheap replacements for specific specialized tasks over GPT-4, and easily outperform GPT-4 for the right kind of tasks. The 4k context limit was a bit of a bummer.
* New tts that to my ears sounds nearly equivalent to Eleven Labs or Play.ht, at one-tenth to one-twentieth the price (with zero monthly commitment). The Eleven Labs Discord is a bit of a bloodbath right now, most of the general chat is just people saying they're switching. (The Play.ht Discord is pretty dead most of the time anyway, so not much new since this morning.) I will say though that it's a bummer that the OpenAI tts doesn't have input streaming, only output streaming, so latency will likely be worse and you'll have to figure out some way to do chunking yourself which is fairly annoying, but for any kind of personalized use case (e.g. a bot talking to customers, as opposed to using pre-recorded snippets) a 10-20x price improvement is worth the extra pain and may be the difference between "neat prototype" and "shippable to production."
Plus, massive price drops for OpenAI's existing products across the board, along with a legal defense fund to protect OpenAI customers from getting sued for using OpenAI models. If you're building an "OpenAI wrapper startup," today was a very good day. If you're competing with OpenAI, though... Oof.
It'll be some pimply intern somewhere that'll blow the lid off things with some ultra-clever-yet-painfully-obvious use case.