Hacker News Reader

Comment by ein0p | Hacker News Reader

On what do you base your intuitions? The way I see it is either they are good universal approximators (like ANNs), and then they’re usable for everything, or they aren’t, and then they’re at best very narrow in application. I’ve yet to see any evidence for the latter.

View on HN

Then why would you choose to use a CNN over a fully connected neural network for visual data? For the same reason you'd choose a KAN over a traditional neural network if you were trying to fit a continuous function that can easily be modeled as a combination of b-splines. Machine learning models aren't magic and their internal function very much determines the kinds of problems to which they are well applied.

My intuitions about KANs and visual data comes from an impression that it would be hard for a decision boundary on visual data to behave nicely if it could only be built from b-splines.

Judging the usefulness of a machine learning architecture is not a matter of determining which architecture will perform the best in all scenarios.

Universal approximation just indicates existence of a suitable sequence, not that it is findable and doesn't apply with finite numbers of neurons.

But MLPs are not good for everything. Where Simulated annealing works better than auto-diff is the classic example that is easier to visualize, at least for me.

Even if the sequence 'exists', finding it is the problem, it doesn't matter if a method can represent an unfindable sequence.

That said, IMHO, MLP vs KAN is probably safer to think of as horses for courses, they are better at different things.

At least with your definition of 'usable' being undefined.

Uninformed response: if two different universal approximators can have different modes/rates of learning, then even if they're usable in theory, they could be less good in practice.

Reply on news.ycombinator.com