Hacker News Reader

Comment by p1esk | Hacker News Reader

What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous largest one (from Nvidia) with more than double the size.

The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to clarify.

Another big problem is we don't have any way to do quality filtering on the outputs. From my experiments with GPT-2, it produces one interesting paragraph of text out of 20 - if you squint at it really hard. And most of those 20 don't make much sense at all.

So no, the existing ideas are definitely not enough. Maybe some novel hybrid of symbolic AI with statistical optimization will lead to a breakthrough. This one does not strike me as anything other than "let's use moar weights!!"

View on HN

Some papers which you might find interesting:

Lin, B. Y., Chen, X., Chen, J., & Ren, X. (2019). KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. 2822–2832. https://doi.org/10.18653/v1/d19-1282

Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2019). K-BERT: Enabling Language Representation with Knowledge Graph. Retrieved from http://arxiv.org/abs/1909.07606

Trinh, T. H., & Le, Q. V. (2019). Do Language Models Have Common Sense? Iclr, 1–12.

Ostendorff, M., Bourgonje, P., Berger, M., Moreno-Schneider, J., Rehm, G., & Gipp, B. (2019). Enriching BERT with Knowledge Graph Embeddings for Document Classification. Retrieved from http://arxiv.org/abs/1909.08402

> they don't have any common sense

What do you mean by this? Of course they do, learned from their training data. For example, here is quote from conversation 38 of https://github.com/google-research/google-research/blob/mast...

Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi.

It seems to me Meena "knows" bulgogi and kimchi are Korean foods. Isn't that common sense? If it isn't, what do you mean by "common sense"?

Try asking it a follow up question not commonly found in the training data: such as "do you think bulgogi would grow on Mars?", and see what kind of gibberish you will get in response. Moreover, the model has no way of self-diagnosing whenever it produces gibberish.

What is the baseline comparison here? If you asked a random 100 people on the street this question?

I'd buy that if Meena could infer and reason about her own answers.

Human: Do you like Korean food in general?

Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi.

Human: Ok what should I shop for ?

Meena : You've got almost everything but you need a pear, the steak and some ginger.

The problem with language models as commonsense is that they are collections of patterns and associations, and that they don't have inference models or solvers - unlike my dog for example!

unlike my dog for example!

A more relevant analogy might be a talking parrot :)

Parrots might have what we call a language model... They definitely have inference and autonomy!

Reply on news.ycombinator.com