1,081 karma · joined January 5, 2017
So it's not that artificial systems cannot, as a category, have models of their environment and the actions they can take within it, this "cargo cult AI" concept comes up when people jump the gun and see those capabilities in much simpler systems, even including ChatGPT-4. And it's disheartening to see narrative on the subject driven more and more by people who have not taken the time to learn about the subject matter, for all the interest they seem to show in talking about it.
It's a mix of things at different levels. Some is high level "here's LangChain or Automatic1111", some is more in-depth paper reads. I covered BLIP2 recently, someone else did PaLM-E two weeks ago, I'm gonna touch on SAM this week.
I'm getting some good code starting points, and talking the idea through step by step with a chatbot is really helping me clarify what i need to do, but I've still got API docs open for the libraries it uses because it likes to make up functions, including the core one on which the project logic hinges. (But that's ok, because now I know I just need to write a function that works that way and i can do that).
Pretty cool and helpful! And I can imagine it getting better with GPT-4 or code-specific tooling. But it's generating value on the order of like.. many other SaaS offerings that have come onto the scene that try to ease pain points in coding workflows. Versus value of the sort that upends society and my entire way of life. A great new tool that I should learn about to make rote bits of my activities faster and easier, a story that's a bit more familiar in tech than some of the more breathless stories about AI make it sound.
I say that as someone who works in the space and loves it tbc. And I fully expect to see some wild stuff make it IRL this decade in both NLP and CV, I just think the rubber hits the road a bit more slowly than pop social media discourse would have one think.
As a decent first-order metric - follow the ratio of companies getting money for using LLMs to do something, to companies getting money for providing LLMs and associated tooling to others. The bigger that ratio gets, the more real world impact LLMs are having.
Google's recent PaLM-E paper shows how much progress is already being made towards exactly this sort of multimodal embedding soup. Things are going to keep getting wilder!
In fact the projection operations are the only learned part of a Transformer's self-attention function -- the rest of self-attention is just a weighted sum of the input vectors, where the weights come from the (scaled) vector correlation matrix.
If you find yourself feeling anxious about AI, spend time learning how modern transformer-based predictive and generative models actually work. From resources rooted in math, statistics, and computer science; not doomer bloggers.
This is consistent with recent rulings about copyright by AI outside the art domain.
I didn't know this before moving here, I don't know if I'd call it _amazing_ weather but it's certainly odd to watch the rest of the country deal with insane heat and cold while SF basically just... stays at 60.
You can quickly get overwhelmed by the million good resources out there so I'll keep it to these three. If you have a strong CS background, they'll take you a long way:
(1) Transformers from Scratch: https://peterbloem.nl/blog/transformers
(2) Attention Is All You Need: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547de...
(3) Formal Algorithms for Transformers: https://arxiv.org/abs/2207.09238
I should figure out how to add sound. It's always inspiring to see the great work of others :)
The generative AI results of the past year straddle the invention/discovery divide -- I've seen snarky internet takes recently along the lines of "why are tech bros solving art and poetry instead of our tedious labor", but what people not clued into the field don't get is that:
(a) This generative stuff is easier specifically because it doesn't have to interact with the real world. It's just data manipulation, there's a ton of it to learn from, and it's basically ok if a lady in a picture has six fingers or if the poem you generated has an incorrect meter on line 8. IRL interactions are harder to get data on, and nobody wants their bulldozer AI to hallucinate that the building next door is part of what needs to be demolished.
(b) These generative tools can exist precisely because there is regular statistical structure in (art/language/music). It takes some work to capture it, but it's the information's structure itself that enables, e.g., cloning a voice from a few seconds of recordings. Making research that exposes that fact taboo just means that actors who exploit it anyway will have a bigger advantage in abusing it.
It can also help to understand where a line of research comes from -- in VALL-E's case, it's an outgrowth of research into neural audio compression, an area with very clear advantages for information technology. If one starts shipping that technology and parts of it can be used for few-shot cloning of others' voices, it seems better that we're all aware of the fact.
In general this sort of communication seems to be a sticking point for the engineering-minded -- it's cool you built a backend in Go, but a backend for what? What's the hook that will draw me in to investing the time to check out your particular tool?