Ah, I think I see what you mean: you are saying that because it's better than humans at predicting the next element in a sequence it's good at generalising. Is that correct, or am I misrepresenting your point?
Basically there are two approaches to sequence prediction.
The traditional style, linear regression, ARIMA, RNNs etc. where you directly predict the next element in a sequence. The output is on the same level of abstraction as the internal values used in the model.
There is also the new-ish style where you predict symbols instead of predicting the values directly. You can predict symbols representing numbers or you can also predict a symbolic formula that can be used to extrapolate the values perfectly. This is the way humans do it.
And my point is that when you look at the symbols embedding they do have interpretable structure that model can use to generalize. And experiments seems to suggest that DNNs models are indeed generalizing.
Have you played around with the demo?
http://recur-env.eba-rm3fchmn.us-east-2.elasticbeanstalk.com...
You can input there a sequence and it will predict the expression and next elements.
For example, for the sequence: [3, 1, 8, 9, 18, 19, 37, 74, 92, 166, 332, 406, 738, 1476]
It predicts: u_n = u_(n−1) + (u_(n−3) (n mod 3)) + u_(n−4)
It's a really decent guess, and if I give it one more element (from 14 to 15 elements) it gets it correctly.
I mean the values of n_pred and n_op in tables 7 and 8. They go up to 10.
I haven't tried the demo. Maybe I'll give it a go if you say it's so good.