To be fair (as one who hasn't paid too much attention to these developements in the last few months/years), the sheer amount of individual facts that are apparently encoded in a fixed nn architecture is what amazes me the most here?
I get that it has 175B variables, but it's not like you can used them in a structured way to store say a knowledge base...