4,157 karma · joined May 28, 2011
Email: <myusername>@gmail.com
I'm not an expert in LLMs though, this is just my understanding of classifiers in general. Maybe with enough data this consideration no longer applies? I'd be interested to know.
It's really a shame that people have made it almost weird to not have cameras on. Admittedly it can be nice to see the other participants in the meeting but I find it's only useful for the first 5 minutes, then it may as well go to audio. Saves on bandwidth too.
does this basically correspond to moving each weight either up or down by a fixed amount? I'm a bit surprised you don't at least need a "stay same" bit, but i suppose it could balance out over multiple iterations.
Interesting that it works at all. Although, thinking on it, I could see it maybe even having a nice regularizing effect where every layer would end up have similar weight magnitudes. (like projecting onto the local n-ball as mentioned in a paper posted recently on HN)
That sentiment goes back pretty far in Facebook's history
[1] https://www.yahoo.com/news/mark-zuckerberg-branded-early-fac...
But, shouldn't that situation be handled somewhat by backtracking sampling techniques like beam search? But maybe that is not used much in practice due to being more expensive.. don't know.
In other words, your complaints are ironically not about what the article is discussing, but about, for better or for worse, attempts to solve it.
For instance, I know this is Canadian so different jurisdiction but here is an example of one: https://montreal.ca/en/services/digital-book-loans
(Jk of course I know what you mean, but you can seriously see text prompts as compressed forms of programming that leverage the model's prior knowledge)
I think for real world application one challenge is going to be the "action" signal which is a necessary component of the conditioning signal that makes the simulation reactive. In video games you can just record the buttons, but for real world scenarios you need difficult and intrusive sensor setups for recording force signals.
(Again for robotics though maybe it's enough to record the motor commands, just that you can't easily record the "motor commands" for humans, for example)
Or is this better thought of as a powerful matplotlib alternative?
So the encoder produces a latent space that is more efficient to train a diffusion model on, since diffusion models use Unet-like architecture that must be run many times for a single inference. The latent space is restricted by a KL penalty to a Gaussian shape such that any sample from that shape will map through the decoder to a high-quality image. This makes the generative job of the diffusion model much easier because it can focus on content and semantics rather than pixel-level details
You can see the two optimisers at work in the AutoencoderKL class in the Stable Diffusion source code here: https://github.com/CompVis/stable-diffusion/blob/main/ldm/mo...
Their still in use literally by every latent diffusion image generator, they typically target the latents of a GAN-trained decoder.
Same for audio, most audio models generate some representation that is converted to audio by a GAN-trained codec.
I get that this must be one aspect of the necessity of the GIL but I mean, C++ also has eager free behavior due to RAII and threads are working fine there, as long as you know what you're doing. Perhaps that's the rub though, it's pretty easy to crash/deadlock in C++ and we blame the programmer rather than the language.
How is this loss calculated though? Since it is called "loss" and not "performance metric", I'm going to assume it is the teacher forced cross entropy loss.
I'm not too familiar with LLM training but having been doing some fair amount of seq2seq training lately in other domains I've observed that the relationship between "loss" and autoregressive inference performance gets very narrow towards the end of training. What I mean is that smaller and smaller reductions in loss lead to better improvements in the autoregressive output. So at least I have some doubt that in practice that 1% loss improvement is not actually incredibly significant with respect to how well the model actually performs at inference time.
But I'm pretty interested in this topic and if people here have observed this or the contrary I'd be curious to know.
What's funny to me is, looking at that invention design, how crazy this guy must have appeared to his peers, like, "look it writes the sound into the ashes!!!". "Sure Eddy, buddy, let's get you a nice cup of tea and calm down.."
Yet he was on to something amazing that would change how we live.
I suppose there was a "crazy inventor" culture at the time though, with so much new understanding of mechanical physics and engineering developing at a such a rapid pace, so maybe it wasn't so out of place, what a time that must have been to be alive..