Hacker News Reader

I'm a researcher in the field. Not in NLP anymore, but I worked on that as well, years ago, and I keep up with the research. You can't "understand the context" of "War and Peace" unless you have real, actual AGI. I doubt actually it can be fully understood at all when translated to English and read by someone without the right cultural background. This is an extreme example, chosen to make it easy to see that it applies to any non-trivial text.

Let's take a question answering example. Take just about any recent deep learning paper and try to answer detailed, higher level questions against it. To use a concrete example, take MobileNet V3 paper and ask your system "do I put activation before or after squeeze and excitation" (correct answer is "before"), or "do I need a bias in squeeze and excitation module" (correct answer is "it depends on the task"). You won't be able to, because a lot of things are just _assumed_, just like in any other realistic example of text written for human consumption. The facts are encoded externally as information about the world, and they're so fine grained and contextual, that we don't even know how to begin incorporating them into the answers, let alone do so contextually and probabilistically, like human mind does.

Maybe. I take a really optimistic view of attention based mechanisms, like I explained and added to my original post. If you read the recent reformer paper, they produce a new way of computation to start building a model that can in some way, encode the relationships between different parts of war and piece. The bottle neck right now is computation, we don’t know how well these models can learn when that bottle neck is removed!

I’m optimistic because I believe that the contextual information that you are describing, is already there in the vast expanse of the internet.

But I will also add, I think none of this will spawn AI, just that it will spawn new technologies that are categorically different.

I think it will merely spawn technologies that are less fragile, yet still too fragile and unsophisticated to be practical for full blown conversational user interface the likes of which you see in the Avengers movies. It will (and already does) make a difference in simpler tasks where you can get away with just counting numbers, such as search/retrieval, simple summarization, constrained dialogue in chatbots, stuff like that.

What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous largest one (from Nvidia) with more than double the size.

The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to clarify.

Another big problem is we don't have any way to do quality filtering on the outputs. From my experiments with GPT-2, it produces one interesting paragraph of text out of 20 - if you squint at it really hard. And most of those 20 don't make much sense at all.

So no, the existing ideas are definitely not enough. Maybe some novel hybrid of symbolic AI with statistical optimization will lead to a breakthrough. This one does not strike me as anything other than "let's use moar weights!!"

> they don't have any common sense

What do you mean by this? Of course they do, learned from their training data. For example, here is quote from conversation 38 of https://github.com/google-research/google-research/blob/mast...

Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi.

It seems to me Meena "knows" bulgogi and kimchi are Korean foods. Isn't that common sense? If it isn't, what do you mean by "common sense"?

Try asking it a follow up question not commonly found in the training data: such as "do you think bulgogi would grow on Mars?", and see what kind of gibberish you will get in response. Moreover, the model has no way of self-diagnosing whenever it produces gibberish.

What is the baseline comparison here? If you asked a random 100 people on the street this question?

I'd buy that if Meena could infer and reason about her own answers.

Human: Do you like Korean food in general?

Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi.

Human: Ok what should I shop for ?

Meena : You've got almost everything but you need a pear, the steak and some ginger.

The problem with language models as commonsense is that they are collections of patterns and associations, and that they don't have inference models or solvers - unlike my dog for example!

unlike my dog for example!

A more relevant analogy might be a talking parrot :)

Parrots might have what we call a language model... They definitely have inference and autonomy!

Some papers which you might find interesting:

Lin, B. Y., Chen, X., Chen, J., & Ren, X. (2019). KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. 2822–2832. https://doi.org/10.18653/v1/d19-1282

Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2019). K-BERT: Enabling Language Representation with Knowledge Graph. Retrieved from http://arxiv.org/abs/1909.07606

Trinh, T. H., & Le, Q. V. (2019). Do Language Models Have Common Sense? Iclr, 1–12.

Ostendorff, M., Bourgonje, P., Berger, M., Moreno-Schneider, J., Rehm, G., & Gipp, B. (2019). Enriching BERT with Knowledge Graph Embeddings for Document Classification. Retrieved from http://arxiv.org/abs/1909.08402

I am sorry, you come across as extremely over-enthusiastic, without too many specifics beyond “we’re just about to figure it all out”, “you just wait”, and “it’s gonna revolutionize everything”. We’ve seen this before with ImageNet, didn’t we? When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn’t it. Transformers are great (well, if you have the compute that is) don’t get me wrong, but let’s not get ahead of ourselves. The field is over-hyped as it is.

I'm not saying this will spawn AI. Just that, like computer vision was essentially solved by CNNs, NLP will be solved by transformer models.

>computer vision was essentially solved by CNNs

That is a rather contentious claim.

To say the least.

> When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn’t it.

I don't think anyone familiar with the area thought that ConvNets will give us AGI.

However, their effect has been huge! It's hard to overstate this. Computer vision used to be a small niche topic, with tons of effort required to get something working even on simple images. The quality of today's ConvNet predictions is way beyond anybody's imagination in around 2010. Models built around that time were like a house of cards. Extremely carefully crafted for specific scenarios, where moving one threshold a bit would destroy your output.

Crucially also, convnets have exceeded human performance on several important vision tasks.

You have to be very careful with such claims. For example it may be able to tell apart tons of dog breeds at a superhuman level, but that's not really what people imagine if they hear such claims.

Also sometimes in medical imaging the conditions are very different from actual practice. For example the doctor may be worse than the convnet on certain types of low-quality, low-dynamic range images that someone preprocessed in a particular way. But sure, in the medical field some error prone, boring counting tasks and spot-the-cancer-in-your 200th-image-today, the machine can perform actually better.

But what tasks specifically do you have in mind?

>Find an answer in a text document that spans multiple non-contiguous paragraphs

>Understand context across a whole book.

to have truly much utility when it comes to context a model doesn't just need to correlate information across some text, it also needs general knowledge and understanding so that it can produce knowledge which is only implicit or not even present in the text itself.

You can make the transformers as large as you want, NLP models still fundamentally suck at answer trivial questions like "If Pete was alive in 2000 and alive in 2050, was he alive in 2025?"

Since you seem to be observing this closely, I'm curious what steps are being taken to identify and avoid bias in the results generated by these systems?