Hacker News Reader

OT: Can you send me a link to your 1999 Elsevier article? The old thread is locked due to age but the topic is very relevant to me...

As to on topic discussion - wouldn't you only need to take out the comms? Then the fellow subs wouldn't know anything was amiss.

Taking down several redundant, military grade links doesn't sound easy, but finding every sub in the ocean seems much harder.

Maybe there is a watchdog / normal coded communication/expected updates, so it would have to be more of a MITM thing. Even in the worst case, it would leave a limited window of time to act. Not to mention the submarine would likely be forced to try and hail "ground control" which would make DF + geo much easier...

You do seem to have a distaste for the term machine learning.

It is applied math, anybody who claims otherwise could not be more wrong. It is applied probability theory and computer science with lot more focus on algorithms than is expected of a statistician, and yes without a fetish for asymptotic normality. Usually, but not always, with a stronger focus on prediction accuracy rather than on accuracy of parameter estimates. In that sense its closer to (non-parametric) decision theory than to run of the mill stats.

Yup. I got all that from Leo Breiman, not his earlier work as an "academic probabilist" but his later work in classification and regression trees (CART), random forests, etc.

Yup, if look carefully, in many common cases can do well on prediction accuracy even when parameter accuracy is not so good. E.g., can even do well on prediction accuracy when parameter accuracy is zilch, e.g., some, but not all, cases of over fitting.

You are right about the applied math: Machine learning has a lot that is good and new, however the good is mostly not new and the new, not very good. E.g., I watched a Prof Ng lecture on the gradient descent unconstrained minimization involved in their fitting, and it seemed that either their minimization problem is awfully easy or Prof Ng didn't know much about the applied math of such minimization going way back.

So, machine learning stole a lot from some now quite classic applied math, assigned some new names, e.g., learning and another anthropomorphic name they dreamed up for iteration step size, and claimed something new. That's called academic theft, plagiarism, or some such, right?

Then there's the hype. So, just from machine learning, the pop culture press is extrapolating to the Terminator movies. Absurd.

Then, bigger, there's the implicit assumption, if only from just playing with the words, that the main path to smarter machines is machine learning. Well, if the effort is limited to Breiman's piecewise linear curve fitting, that path is absurdly limited. Much better for making machines smarter, i.e., do more, work better, is just to start with lots of now classic pure/applied math, get out a clean sheet of paper, and derive what need for the problem at hand. As much as I like Breiman, both his early work and his last work, there're oceans more good work in applied math that is nothing like Breiman's last work, e.g., with high irony, his earlier work!

Uh, here's another case where it looks like a machine has learned -- stochastic optimal control. In some cases, it can look darned smart. Another place -- game theory, to be more specific, the von Neumann-Morgenstern work. Then the Nash generalization. But such things are far from the current machine learning.

As I have mentioned several times to you start with vapnik. That's where modern ml begins. Brieman is great but he got a few things wrong, for example he thought boosting and bagging are fundamentally the same, that is they are just variance minimization methods. In fact boosting us what gives random forest its power. BTW less well known trick, use random rotation not just random selection of features or datapoints. Their's lot more to ml than decision trees. One of its fundamental framework is convergence of empirical processes, which is essentially glivenko cantelli on steroids. Vapnik and Chervonenkis provided the first modern breakthrough result but a lot has happened since then.

I would say screw the hype just read what it is. I think your disdain for the name 'machine learning' is getting in your way.

I strongly disagree with your allegation of academic dishonesty though. Optimization has been a fundamental part of ml from the very start and with a strong overlap in the community. I am sure nemirovskii will ring a bell. In any case how can ml steal from applied math when it is applied math. The lingo differs here and there: estimating parameters become learning parameters etc.

Andrew NGO's course is not for you. Go directly to the books and then conference and journal papers.

Focus varies but info theory, stats, signal processing, approximation theory are pretty much the same thing, so are their open problems.

Btw you could not be more wrong about stochastic optimal control or repeated games being far from current ml. Oh and another major major tool in ml is geometry of function spaces. You are reading the wrong sources, probably the populist ones. Get to the real stuff given what i know of your taste in math i think you will enjoy it.

http://amlbook.com

https://www.youtube.com/watch?v=mbyG85GZ0PI (incidentally for graycat, Yaser Abumostafa is a Muslim Egyptian immigrant from Cairo) He covers the VC dimension in https://www.youtube.com/watch?v=Dc0sr0kdBVI, and leaves the proof to an appendix of the book.

Thanks. I looked a few minutes at two of his lectures. I'll keep the URL and watch his lectures during dinners.

Lucky he got out of Egypt before they strung him up! Such good people is what US immigration has for some decades now tried to be about. Maybe we will get back to it.

"Muslim"? I don't care if he is Zoroastrian either. Or worships some sun god. I don't care about his religion. I do care if he wants to blow up buildings. Somehow I doubt if he does.

Looking at his videos, my first cut, crude guess is that he is looking at modern generalizations of old discriminate analysis. Yup, that can be important. Maybe it could be important for, say, one of my old interests, anomaly detection, say, as a doable alternative to Neyman-Pearson where often in practice we don't have nearly enough data. Maybe his interest is in medical diagnosis which, IIRC, was some of Breiman's interest.

But, first cut, it looks like, again, the criterion will be, does the model fit the data well? That is, we have little or nothing to recommend the model except that it fits the data well. But, then, in the case of his lectures, it looks like maybe he is making progress to also knowing that the model will predict well. I'm looking forward to how he does that.

In contrast, if that is important, in my work in anomaly detection, discussed here on HN often enough, I found false alarm rate from some derivations in applied probability with no model fitting at all. Okay, I don't care if the cat is black or white as long as it catches mice.

From a glance, it looks like he is addressing what is meant by learning -- terrific! Not just throwing words around! Then he seems to be addressing when such learning is feasible, etc. Sounds good; I've wondered some about something like that.

But my interest now in what he is doing is a bit limited since the core math in my startup seems to be quite different.

Thanks.

> it looks like maybe he is making progress to also knowing that the model will predict well. I'm looking forward to how he does that.

Yes that's exactly it. By way of Vapnik and Chervonenkis' result (essentially an uniform law of large numbers) one upper bounds the expected accuracy (over the unknown distribution) of a classifier in terms of the training error and another quantity that depends on the class of hypothesis that one is using. One can give bounds even when one is using an infinite class, for example all linear functions in the feature space, or some Hilbert space of functions.

This was one of _the_ major early break through result. Its often quoted in the context of ML but it really is a result in probability theory. Since it bounds the most pessimistic situation possible, they are quite bad (although achievable).

It also brought about a paradigm change in the mindset. Since the optimal classifier is just the thresholded conditional density, early approaches had focused mostly on estimating this conditional density. But that's an impossible task. V&C showed even if you do not have enough data to learn the density, you may have more than enough for good prediction accuracy. Don't learn the conditional density, just learn the discriminating function directly by optimizing its expected loss.

People have moved to different tools to bound expected prediction accuracy. You get a lot more reasonable bounds, say with the PAC-Bayesian theorem.

Key thing is that these are distribution independent, non-asymptotic and also dimensionality independent.

Thanks, I'll keep that.

"glivenko cantelli on steroids", good. Sounds like they actually did something.

Yes, I'm torqued by the new learning labels on old bottles of pure/applied math, but that is not in my way.

> The lingo differs here and there: estimating parameters become learning parameters etc.

Rubs my fur the wrong way.

If they have some stuff beyond borrowing from Breiman, okay.

What's "in my way" now is my startup: I've got the math derived and typed into TeX and the 80,000 lines of typing for the code, with the code running, intended for production, and in alpha test, so just for now I no longer have any pressing math problems to solve.

But, in time, I may return to my math and tweak it a little to try to get some variance reduction. Maybe some of the better machine learning literature would help, or maybe I'll just derive it myself again.

Function space geometry is about where much of my core math is.

Thanks.

Heh! indeed, its all geometry :)

Happy to hear back from you. I am actually gladdened that your anomaly detection work is getting some interest on HN lately. Hope something comes out of it. I am now slowly coming to the conclusion that pushing better methods on an existing stack would be really hard. Too much friction, too much politics. Perhaps the way is to create your own better cloud of servers, but that's really big league stuff. Not sure I have the stomach for that.

Curious if you have given any thought to the choice of the metric space where you define your statistics. That might play an important role from what I have seen. There might be an interesting manifold story there.

Big spoilsports are non-stationarities and even bigger are those fat tails. If only everything had a moment generating function.

I see that you have been pointed to Abu-Mostafa he is definitely a good source. Not that Andrew Ng is unaware of the stuff, far from it, he is fighting a different battle: to make parts of ML a commodity.

If you have time then you can browse the proceedings of COLT (conference on learning theory) and ICML.

> or maybe I'll just derive it myself again.

You almost always have to derive it yourself anyway even after you have seen the derivation by somebody else.

This was around https://en.wikipedia.org/wiki/SOSUS and better stuff is around today.

Nothing like a big bang to scare the population into projectfear.

> If they track it for very long, then that is provocative.

Both sides spent the entire Cold War shadowing each other constantly. There was nothing provocative about it.

I question if the Soviets were able to shadow a US SSBN. I shouldn't give more info.