Hacker News Reader

Comment by sydthrowaway | Hacker News Reader

What causes this in a net? I’m guessing the RNN gets in a catastrophic state..

I would say overfitting - the net doesn't "understand" the code in any meaningful sense. It just finds fitting examples and jumbles them a bit.

Understanding would mean to have an internal representation related to the intention of the user, the expected behavior, and say the AST of the code. My pessimistic interpretation of this and many other recent AI applications is that it is a "better markov chain".

a markov chain can have an internal representation related to the intention of the user. I guess this example just got copied a lot and is therefore included multiple times in the training data, forcing the network to memorise it. Neural networks always memorise things that appear too frequent. Memorized Artifacts in an otherwise working neural network is usually seen as a "bug" (since the training allowed the network to cheat), not as a proof that the network didn't generalise.

This is the network working as designed.

I mean, if you wrote an autocomplete system for written english and asked it to complete the sentence "O Romeo, Romeo" what would you expect to happen?

You'd expect it to complete to "O Romeo, Romeo, wherefore art thou Romeo?" - a very famous quote.

How else could you produce the single right output for that unique input, other than memorising and regurgitating?

> You'd expect it to complete to "O Romeo, Romeo, wherefore art thou Romeo?" - a very famous quote.

What about completing it to "O Romeo, Romeo, brave Mercutio is dead", based on the context, as advertised?

Right, but the demonstration gave zero context and came up with the original function. It would have been interesting if it were instructed to produce the function in Haskell or some other programming model.

> catastrophic state

No, overfitting is the normal state for neural nets.

Neural nets aren't magic. You actually need quite a bit of complexity and modeling of interrelated problem spaces to get anything more than a childlike naivete or trauma savant-like mastery of one particular area with crippling deficiencies elsewhere.

Reply on news.ycombinator.com