291 karma · joined June 18, 2019
I don’t know how much tool use there is these days of the llm „just“ calling image generation models, with a bunch of prompt reformulation for the text-to-image model which is most likely a „steerable“ diffusion model (really nice talks by Stefano Ermon on youtube!).
Actually multimodal models usually have a vision encoder submodel that translates image patches into tokens and then the pretrained llm and vision model are jointly finetuned. I think reading the reports about gemma or kimi VL will give a good idea here.
Looks like big and small tech are mostly doing things wrong then?
Turns out this article _is_ the one from NYU, just on arxiv now:
https://graphdeeplearning.github.io/post/transformers-are-gn...
[0] e.g. https://arxiv.org/abs/2003.10536, [1] ProteinMPNN, Dauparas et al.
Google announced they are serving 500T tokens per month. State of the art models are currently trained with less than 30T tokens. Even if training tokens are more costly to run (eg, a factor of 3x for forward, backward, and weight updates, and take another factor of 2x for missing quantization), you end up in a situation where inference compute dominates training after a very short time of amortization.
I‘ve heard people say that GRPO giving a zero gradient in cases where neither the current sample nor the group scores give any reward is advantageous for optimization. It avoids killing your base model with low signal to noise updates, that can be a problem in PPO where the critic usually causes a non-zero gradient even for samples where one would rather be like „problem too hard for now, skip.“
I‘d be curious to hear you lay out your thoughts though!
I can say with certainty that I wouldn’t know how to wield matplotlib and pandas with such fluency in an hour, even though I am perfectly able to read the implementation and query for some relevant intermediate results to check my mental model.
Granted this is not the world’s most complex problem, but that is a good example of the domains where these tools are incredibly useful and productive already (I didn’t have to consult the docs even once). So in a way I think of LLMs as very good interfaces to the docs :)
I often feel that the UI aspect of new technologies is underappreciated. All our computers (even grep) are turing complete. This means software engineering is fundamentally a discipline of building better user interfaces that allow us to do whatever we want more easily.
I am always curious how other people experience these things as so useless :)
What makes you think that it is „bad“?
There is this related saying that „the fastest marathoners burn just as much calories as the slowest to complete 42km, they just do it in a much shorter time“. This kind of range is not available to walking.
It shows that in contrast to most animals, the efficiency curve of humans for running speeds is extremely flat, ie, we are about equally efficient at many different speeds, while the kind of game that we hunted was not.
The discrepancy allowed us to find a speed where we could exhaust the animal after 10-30km (as I understand) provided we were also excellent trackers.
https://www.nature.com/articles/nature03052.epdf?sharing_tok...