Hacker News Reader

Comment by visarga | Hacker News Reader

How come you know the efficient-transformers family, when I ask questions about transformers in ML interviews nobody has heard of them. Can't figure out why it's not common knowledge. For years all the transformer papers were about reducing O( N^2 )

View on HN

Two reasons: They were only recently implemented in large production models and are not part of ye-standard-ML-coursera. And I mean, for half the papers claiming that their particular efficiency variant reduces O(n^2) to whatever without performance loss, we found that in practice it ain't quite so shiny.

Anyone who has been for whatever reason reading the papers since 2017 has invariably read dozens of these papers.

Anyone who has heard of GPT-x in 202x and started from there probably didn't.

This will likely change with implementation of memory retrieval, some form of linear attention etc. in many productions models, and the democratization of some decoder models... although I have been thinking this for a while.

Don't get me wrong, you want to hire the people who know these papers, especially if they started after 2017 :-)

I have a team of bioinformaticians with very little ML knowledge, but they know of these basic papers... So yeah, if someone claims to be in ML and aren't aware of this foundational knowledge... they're not to be taken seriously.

To be fair ML is (used to be?) pretty broad, so unless someone is actively keeping up with the sota in the high-data sequence modeling area it's quite possible to miss. I know ML teams which were entirely made up of OSML practicioners, because that was the most commonly useful until recently.

The reason they don't know them is they're not serious researchers or practitioners of ML - it's as simple as that. Anyone in this area should have this exceedingly basic common knowledge.

Why learn something noone is using.

Oh I didn't test their in-depth knowledge, just checking to know if they were aware people were writing hundreds of papers on this topic, and what impact would have a sub-quadratic attention that works as well as the regular one. Fishing to see if they remember at least one approach or two.

Many people are using it, but people that are relying on OpenAI to do all of the work for them aren't looking deep enough to realize it.

Can you point to a single major project that does? AFAIK it always leads significant regressions in real world use cases, which is why no major repo uses them.

Reply on news.ycombinator.com