4,157 karma · joined May 28, 2011
Email: <myusername>@gmail.com
Why should I care if some 3rd party is the middleman that builds the system that performs the transformation? I put out my code so that people can use it. The existence of said 3rd party doesn't restrict that in the slightest, in fact it enhances the reusability of my work which seems like a win-win.
What they are not doing, as far as I can tell, at least not intentionally, is just copying code and removing the license and attribution. Even if you don't see the AI output as legitimately "creative", it's certainly transforming and remixing existing solutions in ways that are transformative enough not to be just copies of anything except the most bog standard boilerplate, which is usually not a copyright issue.
People seem upset because there is money to be made, if there was no money involved I don't think anyone would see any issue here.
It's not about the greater good, and I don't mean to be an apologist, but I just legitimately don't see a problem with doing it, or how it's not fair use. It really does seem like fair use to me.
Maybe not but it's certainly worth worrying that this is what will happen just because the line has to be drawn somewhere and that's an easier way to draw it. Baby, bathwater, etc.
The fact that we haven't adopted something much simpler to just be able to express italics or whatever, like markdown, is just bonkers to me. It just shows how little anyone cares about actually improving the situation. And all just to cater to the bizarre corporate need to put logos and banners everything. HTML email is just ridiculous.
I find it a bit difficult to find resources describing the properties of various options for this topic of discrete choices and clustering, apart from a few papers & blogs describing the idea.
Adding scaled unit gaussian noise to the logits
noise = torch.randn_like(logits)*F.softplus(noise_logits)
noisy_logits = logits + noise
Question, if you changed this Gaussian normal for Gumbel noise you would get something like Gumbel softmax, right? I'm curious why not use it? Isn't it a usual way to implement differentiable discrete selection? My curiosity is about the effectiveness of Gumbel softmax since I have had some trouble using it in practice so I'm curious why it's not used here and if there are downsides to it compared to other methods. Honestly just adding normal noise like this seems simpler anyway.Heh, this gives me the idea of training an image generator not on tokens but on rendered text of the prompt in some 8x8 font, that would be a fun experiment ;)
So practically speaking it's a bit more complicated to calculate how much the model can "think". Of course once a token is output it is committed to that (in the most basic scenario), but that doesn't mean it is not still "thinking" as it produces subsequent tokens.
> perhaps a notepad
Exactly, the context and previously output tokens can be considered such a notepad since they are input for the next steps of the model.
And that's why the concept of generalization is so important on machine learning, and as a consequence, why the internal representation of that "lookup" matters.
By definition a lookup table can only store data it is given. However, the idea of ML systems is actually to predict values of inputs that are similar to but not given in their training data.
Interpolation and extrapolation, key components to applying ML systems to new data and therefore critical for actual usage, are enabled by internal representations that allow for modeling the space between and around data points. It so happens that multilayer neural networks accomplish this by general and smoothed (due to regularization tricks and inductive biases) iterative warpings of the representation (embedding) space.
Due to the manifold hypothesis, we can interpret this as determining underlying and semantically meaningful subspaces, and unfolding them to perform generalized operations such as logical manipulations and drawing classification boundaries in some relatively smooth semantic space, then refolding things to drive some output representation (pixels, classes, etc.)
Another view on this is that these manipulations allow a kind of compression by optimizing the representation to make manipulations easier, in other words they re-express the data in a form that allows algorithmic evaluation of some input program. This gives the chance of modeling intrinsic relationships such as infinite sequences as vector programs. (Here I mean things like mathematical recursions, etc.) When this is accomplished, and it happens due to the pressure to optimally compress data, you could say that "understanding" emerges, and the result is a program that extrapolates to unseen values of such sequences. At this point you could say that while the input-output relationship is like a lookup table, functionally it is not the same thing because the need to compress these input-output relationships has led to some representation which allows for extrapolation, aka "intelligence" by some definitions.
The fact that these systems are still very dumb sometimes is simply due to not developing these representations as well as we would like them to, for a variety of reasons. But theoretically this is the idea behind why emergence might occur in an NN but not in a lookup table.
It's true that bias questions often come up in ML context because fundamentally these algorithms do not work without data, but _all_ algorithms are designed by people, and _many_ can involve data in setting their parameters. Both of which can be sources of bias. ML is more known for it, I believe, because the _inductive_ biases are less than in traditional algorithms, and therefore are more keen to adopt biases present in the dataset.
https://www.tandfonline.com/doi/abs/10.1076/jnmr.32.4.369.18...
Although that idea was more about using haptics as a medium to compose explicitly for rather than translate existing music to.
I think we too easily forget about all the other reasons for fine tuning. Without it I assume using ChatGPT would be more like chatting with an average reddit comment or Twitter post, full of typos and emojis and random gif links.
I'd much rather chat with something that gives me reasonably "cleaned up" answers even if it takes a little work to get it to do what you want.
It's obviously rather hard to hit that fine-tuning sweet spot though, no doubt.
So it's strange to me to claim that it doesn't have an abstract representation.
But, maybe the latent space of a diffusion-VQVAE pipeline is fundamentally different from that of JEPA, I haven't read the relevant papers for that. Curious if someone could explain if they are different ideas of representation.