sendtown_expwy

I have to admit that when I read the intro header of the piece on the page, “They made chemistry choose a mirror image,” it struck me as LLM-speak and I immediately lost interest. I have no idea whether the authors used one to draft that phrase, which seems like an innocent summary of their contributions, but somehow I feel like if it were three years ago we would have never seen that in the title.

This post would be more effective if Mustafa treated it like what it is: a position paper, saying that for our benefit, it’s better we interpret LLMs as such. But it sounds like he just doesn’t understand it’s a non-falsifiable claim, and his asserting of it makes it sound paternalistic.

Meta offering the highest salary could be because people actually do care about this, and so Meta needs to pay more for people to think it’s worthwhile working there.

That's the point. 100 parameters is sufficient to overfit, and it's a number that's independent of the input size. Do you have a reference for your statement?

You are incorrect about the input dimensionality mattering. Let's say you have 100 high-res images with yes/no labels. If you hash the images and put their labels in a hashmap, you can say this is a "learned" function of 100 parameters which achieves zero training error on the dataset. This parameter count is independent of input dimension. Why do you think this would change when this mapping is replaced by a smooth neural network mapping?

GPT is trained to predict the input (estimating p(x)), versus predicting a label given an input (p(y|x)). So in the case of GPT you can use the input dimensionality as a "label", as another responder has mentioned. ImageNet classification is different (excepting recent semi-supervised or unsupervised approaches to image recognition).

The ability to generalize in the typical imagenet setting is, as the article says, a byproduct of SGD with early stopping, which in practice limits the number of functions a deep neural network can express (something not considered in an analysis which only considers parameter count).

"Fortnite will be blacklisted from the Apple ecosystem until the exhaustion of all court appeals"

It's not about sticking it to them. It's to increase the cost of their appeals and pressure them to drop the case.

What, you want people to host services for you for free?

This is a really great piece. While it's unfortunate this phenomenon occurs, in a way it's also an opportunity for the many talented female VCs in the industry. I hope they capitalize on it!

I would guess that an average FAANG ML engineer could code up and successfully execute a forward/backward pass on a GPT-1 or GPT-2 model with a day of effort or less. (GPT-3 a little harder, but not significantly). But is that model actually going to perform well? Most likely no. Model performance varies significantly due to subtle details in data processing implementations, seemingly insignificant details in code, and even from different numerical methods of calculating the same semantics.

If you don't believe me, consider that many ML researchers track their commits (or exact code versions) extremely carefully, because oftentimes they will make some change (or changes) they think are inconsequential and later find that actually, their model broke. If they made too many changes, whoops, guess you have to binary search over the diff to see what happened since your last "good run".

If the people who spent months (if not years) tuning a model can't tell whether it will work from the code, how could anyone else? Most ML researchers will not bother with most code that doesn't give proof of results (in terms of a model that can actually be evaluated) because it is just so unlikely that it will actually work well. Now, it might "work" in the sense that it converges and does something when you prompt it with examples. But will this GPT-3 reimplementation actually outperform say, the 10x smaller T5 checkpoint that was released by Google, or the other smaller language models others have released? If it doesn't, it's hard to argue that its very useful at all.

I think that's the spirit of why the original commenter said what they did, but I still do applaud the efforts of this team (and hope that their implementation is, in fact, highly performant!)

I like this idea, but is the technical implementation of a progressive social policy (whether it be a tax on equity versus something else) actually the hard part?

Alternative take: the hard part is that the US is heterogeneous and people just don't trust each other to not abuse benefits. (You could also say it's racist). How could we give equity to every person when we can't even seem to agree that they deserve basic healthcare?

I am unfamiliar with the term mathematical engineering. What does it even mean?

If its deep in the network, though, couldn't you interpret it as a single neuron responsible for the substring "spider" in the caption? That seems less surprising. But, I see your point.

If the text encoder's representation of "spiderman" and "spider" are close together, then the image encoder is penalized if it makes the image representations far apart from each other. So what we are witnessing here could simply be the neuron that says "I think 'spider' appears in the caption'". Of course it could be that the network understands the metaphorical relationship of spiderman to spiders, but the simpler explanation seems more plausible to me.

The model was trained to match an image with its caption. So its unsurprising to find a neuron that fires across spider man the character and a spider: those are both instances of valid captions. Same with the "iPod" example. Seems a stretch to suggest equivalence to biological neurons (unless you think those are also trained by text supervision. Which is an open hypothesis I suppose.)

This is a wonderful post -- reminds me of how important the hiring processes in the tech community are, and how exciting and amazing it can be for the joy of programming to spread to more places in the world!

It seems like Robinhood, when it stopped trading on GME, decided to literally transfer money from the WSB retail traders to the funds who were short on the stock. But if Robinhood needed to halt trading in order to just survive as a brokerage, (and could demonstrate so), would that justify its behavior? Does anyone know more about laws and regulations concerning brokerages here?

It seems unlikely the model has seen "baby daikon radishes in tutus walking dogs," or cubes made out of porcupine textures, or any other number of examples the post gives.