quantombone

A VR system that lets you walk around your house will have to detect the tables/chairs/walls in your living space and place some kind of digital content in their place so you don’t bump into stuff. VR without perception won’t work. That kind of #VR is essentially #MixedReality

Using PyTorch (and the broader space of machine learning algorithms under the “deep learning” category) really makes me feel like a wizard. But the downside to being Python-dependent is that putting PyTorch stuff into products not easy. I hope PyTorch 1.0 will change that.

Note: This article was not written with Machine Learning in mind, and I will have to re-read the article to better articulate my thoughts on “Machine Learning Wizardry” and juxtapose my own ideas with those of the article author.

Kudos to author: The article’s main metaphor is excellent because it got my creative juices flowing (i.e., brain working at 110% for a few brief moments).

SVMs have convex objective functions, but when people use SVMs, they are using some kind of features + SVMs on top. The success of the approach is both good features, and waiting long enough for the optimizer to converge.

With DNNs, people learn everything (features + decision boundary), and this problem is not convex. Surprisingly DNNs work quite well in practice even though we were taught to be afraid of non-convex problems in grad school around 2005.

If back in early 2000s, we stopped worrying about theoretical issues and explored more approaches like ConvNets, we might have had the deep learning revolution 10 years earlier.

The hinge-loss and the primal form of the SVM objective is really easy to understand. Every ML 101 class would jump into the dual formulation, talk about kernels, RKHS, and all the fancy stuff.

Once you realize that a linear SVM isn’t very different from logistic regression, it starts to all make sense (at least it did for me).

Key insight of the hinge-loss: once something is classified correctly beyond the margin, it incurs a loss of zero.

Now, Something fun to think about. Draw the hinge loss. Now draw the ReLU (which is found all over the place in CNNs). Now thing about L1-regularization (which was used to induce sparsity in compressed sensing). They are more similar in form than you would think.

Back in around 2008, SVMs were all the rage in computer vision. We would use hand designed visual features and then a linear SVM on top. That was how object detectors were built (remember DPM?)

Funny how SVMs are just max-margin loss functions and we just took for granted that you needed domain expertise to craft features like HOG/SIFT by hand.

By 2018, we use ConvNets to learn BOTH the features and the classifier. In fact, it’s hard to separate where the features end and the classifier begins (in a modern CNN).

Here is a zooming example. I definitely noticed that it makes people's eyes look evil. Maybe it's hallucinating animal eyes on top of human eyes...

http://imgur.com/kThM9vf

ConvNets have gotten popular because of their strong empirical results. All the recent work on visualizing CNNs suggests that the community working on Deep Learning still has a lot to learn about their own algorithms.

But high-level notions like a Jaguar is a cat-like animal aren't necessary to perform well on an N-way classification task like ImageNet.

What's more important to note is everybody knows there's plenty wrong with a pure appearance-based approach like CNNs. Every few years a new approach pops up that is based on ontologies, an approach inspired by Plato, etc, but these systems require a lot of time and effort. More importantly, they don't perform as well on large-scale benchmarks. In the publish-or-perish world, you can jump on the CNN bandwagon or start reading Aristotle's metaphysics and never earn your PhD.

Is there an official list somewhere which assigns animals to kinds people you will find in a startup? Like "shark" for VC, "owl" for the PhD, grizzly bear for the bearded Linux hacker, etc?

I remember a political satire from TV in the post-communist days in Poland (1991?) -- Lech Walesa was a Lion. I'd love to see a cartoon/puppets show about startups and/or software dev with animals. The honey badger would def be in there. :-)

I use a pen when I read. I underline important bits and actually write down the key points in the margin. Writing these things down helps memory and I don't really care about leaving the pages in the same state that I found them. Also I don't always have a notebook with me, and I do this to my own books only. I just don't care about "ruining" a book -- I want the knowledge!

Of course you might not want to borrow some of my books when I leave them in this state -- they look like study aids from a med school study-a-thon. But whatever trick YOU find that helps you pound knowledge into your brain is worth it.

Thanks!

Yes to @eof's response. He does know me well, and should be able to clarify in my absence.

There is going to be an AI engine that we use in the future in a similar way that we use Linux today. I mean that "future Linux-ish AI engine" and unless Linus has been busy ML-ing, it won't be called Linux.

You're correct. I really should have used a word like "invaluable". I love my Duda&Hart. I'm not sure why "infamous" felt appropriate. Thanks for the catch.

Thanks, will check out quora links. Follow up question: is it advisable to get application feedback from other YC companies? In my head this only makes sense if the product made by applicant can be used by the startup you want feedback from, but not sure if this is the best kind of feedback.

It would be great to see examples of successful applications, and why they stood out. Seeing how the initial presentation of the idea is different than the company today would help applicants be bolder. There's the Dropbox application which is used as an example, but are there more examples?

I remember taking Moore's grad level Machine Learning course in 2005 (was co-taught with Tom Mitchell). I have great memories from that course, especially interactions with my fellow robotics friends: Mark Palatucci (now a co-founder and officer at Anki), Geoffrey Hollinger (now a professor at Oregon State), and Jon Huang (now a postdoc at Stanford).

Moore gave us a very intuitive way to understand machine learning algorithms, and Mitchell gave us a more theoretical account. I remember working with Jon Huang on a Latent Dirichelet Allocation algorithm implementation for the final project. It was a lot of fun and very intense at the same time. Such great memories!

Moore have me an excellent foundation of concepts, and I'm really glad for CMU to have such a respectable figure back on campus.

I expect great things for CMU, especially Robotics and Machine Learning!

While vision might not be the best avenue for studying general purpose intelligence, it is worthwhile to think of vision as more related to hard AI and less of a problem of applied machine learning.

I think once vision researchers understand just how close they are to hardcore AI, they will realize that there is a plethora of knowledge (not all visual) about the world that can be used to create better object recognition systems.

-Tomasz (blog post author)

I agree that NLP is no less AI than vision. What I wanted to point out in my post (I am the author) is that too many smart computer vision researchers don't view vision as AI at all. Somehow many people's research program is centered around the assumption/desire that a pure image classification approach will solve vision.