Hacker News Reader

Do we even have a good definition of what it means to generalize? I keep seeing this term thrown about and while it makes intuitive sense, and certainly train/test/validation splits are useful, I sort of question the whole premise. I mean, when we consider all possible sets, all optimization problems perform equally well (or poorly), so in the most general sense, generalization is impossible. Which means we have to restrict our sets somehow, but to do that we have to have some measurement as to how they relate, but how do you know in advance? In a sense what the machine is learning is a distance metric between these sets, but the only way to know it's working is to actually run it. To say in advance "well, this machine generalizes this well on this class of problems" seems like an awful stretch.

So much of what we mean when we say "this model generalizes well" seems like baked-in human assumptions about the data distributions that may not really have any basis in reality.

Generalize does need to be more specific in order to make sense. In current ML work, generalizability means something else than what it typically used to mean in AI work.

One way to look at what generalizability is would be like this: problem domains typically come with sets of axioms. Generalizability is the ability for a solution approach to work across different domains that don't have 100% axiom overlap. The wider the difference between axiom sets for a domain, the more difficult / impressive the generalization.

Solving two problems within the same domain, necessarily sharing a 100% overlapping axiom set, is not generalization at all.

The reason the axioms supporting the domain matter is that they (in part) guide which heuristics work, and how they should be applied. And that's the sort of generalizability that is missing from current work: some solutions can make some pre-programmed choices about applying different heuristics depending on the problem set (driverless cars are doing this now). This is the "bag of tricks" approach. But they don't typically morph in how they are applied, or the end that they're trying to accomplish when they're applied.

I have pondered this and have decided what my standards are. I realize I don't have the authority to set those standards for everyone.

GAI is, to me, when machine is able to be given a problem and then, without prompting, decides which data to consume to learn how to solve that problem. It could be told to optimize an automotive design for a 5% efficiency increase without a loss of safety features and while keeping the performance the same - and then go out and figure out what data it needs to learn so that it can solve that task. It would assemble and process that data and then come up with the answer, which might just be that it is impossible with current tech and here is what is needed and this is how to do it.

That's rather verbose and I'm absolutely not the person who gets to define it. But, when someone says AI, that is how I think of it. More so when they say general AI.

That's the frame problem, which is a huge issue for AI. In general it is impossible to solve due to the no free lunch theorem.

Thank you. Would you know where I can look for more derailed info?

A couple links: https://thebestschools.org/magazine/limits-of-modern-ai/

https://plato.stanford.edu/entries/frame-problem/

Thanks. Those will keep me busy for a while.

It's just curiosity, I'm not expecting to enter the field of AI research. It's still fun to learn. Again, thanks.

> And I imagine we'll be arguing that's not 'REAL' intelligence even afterwards.

When Data walks out of a lab, do you really think people will argue that he's not intelligent in the generalizable sense that humans are? Anyone who has watched ST:Next Generation agrees that he is.

> The achievements we're regularly producing now a days dwarf anything those speaking of chess=intelligence times could have even imagined.

I'm pretty sure the AI founders like Marvin Minsky could imagine Artificial Intelligence being generalized over various domains. I forget which one predicted the by the year 2000, an AI would be able to read all the world's books in an intelligent manner. McCarthy assigned a student to solving the robot vision problem over a summer back in the 60s.

Problem is that they underestimated the difficulties in getting a machine to perform tasks that are simple for us, while thinking that tasks like superhuman chess would be the difficult challenges. They had it backwards.

To add to your excellent point; it takes 10-20 years to fully train a human from useless baby to fully functional. And that time is filled with days of continuous exposure to data.

There is no problem domain that humans have been shown to hold an unshakable advantage. I used to be able to say Go but that is no longer even a little bit true. It could well just be time and data from here.

Programming, philosophizing, holding a conversation, playing a sport, applying concepts learned in one domain to another, and a common sense understanding of the world (Minsky's big one).

And it's not just 10-20 years of a blank slate being exposed to data. It's evolved brain structures that know how to learn those skills. That's why a young human child quickly surpasses the learning abilities of chimpanzee.

It's natural to claim that's not generalizing since it's in such a specific domain as well as the fact that it had access to its score.

Yeah. I would like to see the same approach applied to NES games[0] with the capability of beating a game like Zelda or Ultima IV with no domain-specific knowledge. It's not going to happen. These games assume the player knows about human culture, values, virtues such as courage and triumph over the forces of evil. Symbols such as the sword and shield, the heart; they have deep meaning even to a person who has never played a video game before.

To build an AI that can play a game like that, without any specific knowledge of the game itself, would be to build something that likely reduces to general intelligence, though I hesitate to claim it forcefully.

[0] I am aware of this: http://www.cs.cmu.edu/~tom7/mario/

Deepmind also failed to reach human level on a a third of the games they trained it on. And I wonder about the superhuman level. Superhuman compared to average gamer, or really dedicated ones? Because there are people who have played perfect games of Pacman and Pitfall.