HNHacker News
TopNewBestAskShowJobs

sapphireblue

388 karma · joined April 13, 2016

submissionscomments
sapphireblue··on “CPUs are optimized for video games”
Because CPUs are designed around benchmarks representative of real workloads people are running on their computers. Naturally, multimedia and games are a large part of these workloads, this is what drives SIMD adoption.
sapphireblue··on “CPUs are optimized for video games”
This may be an unpopular opinion, but I find it completely fine and reasonable that CPUs are optimized for games and weakly optimized for crypto, because games are what people want.

Sometimes I can't help but wonder how the world where there is no need to spend endless billions on "cybersecurity", "infosec" would look like. Perhaps these billions would be used to create more value for the people. I find it insane that so much money and manpower is spent on scrambling the data to "secure" it from vandal-ish script kiddies (sometimes hired by governments), there is definitely something unhealthy about it.

sapphireblue··on “CPUs are optimized for video games”
C# is just a glue language, physics is run inside the engine as optimized C++, it is vectorized where possible.
sapphireblue··on AI’s Language Problem
You have some good points, but this naive approach of handcoding "cognitive modules" was tried many times in 20th century, and it didn't work at all.

But look at what Deepmind does: it takes these ideas (and also ideas from systems neuroscience), implements them as differentiable modules and trains them on data in end-to-end fashion. This works really well.

Learning is very important, much more important than architecture. If you have a model that can learn you can add more structure later - again this is what modern deep learning is all about.

sapphireblue··on AI’s Language Problem
This is not a question of "consciousness", this is a question of Reinforcement Learning.
sapphireblue··on AI’s Language Problem
I agree to your points. Your comment is a quality one, I mostly talked about other ones.

>Turing-completeness is quite broad and nonspecific, like I said.

It is, but feedforward models (and almost every Bayesian/statistical model) don't possess it even in theory, while RNNs do.

>Doing "some computation" is an obvious statement that doesn't add any information.

Let me be more specific: currently researchers think that later stages of CNNs do something that is more interpretable as computation than as mere pattern matching. Our world doesn't require 50-level hierarchy, but resnets with 50+ layers do good, looks like because they learn some non-trivial computation.

>the jury is still out on whether any of those RNN approaches will be the needed breakthrough.

Sure, we'll see. Maybe there won't be need in any breakthrough, just incremental improvement of models. And even current models when scaled up to next-gen hardware (see nervana) can surprise us again with their performance.

sapphireblue··on AI’s Language Problem
IMHO It is unproductive to rigidly split problems into "merely requiring algorithmic solution" and "AI-complete".

Even with language there is a whole spectrum of language skill. Some animals like parrots, crows, great apes, and then people can learn language at various levels.

Some deep learning models can already learn basic language skills too. The question is, how far can these techniques go. Maybe, pretty far.

sapphireblue··on AI’s Language Problem
>Are the image-recognition AI's described in the article capable of selective attention?

Yes they are: https://indico.io/blog/sequence-modeling-neural-networks-par... https://github.com/harvardnlp/seq2seq-attn

sapphireblue··on AI’s Language Problem
It is already succeeding on language tasks, see https://research.facebook.com/research/babi/

It is funny how every AI post on HN turns into a speculative discussion forum full of words "I think", "likely", "I suspect", "My guess" etc, when all the research is available for free and everyone is free to download and read it to get a real understanding of what's going on in the field.

>what I see is the fact that networks have huge capacity to fit to data and are deep (rely on a hierarchy of features).

Actually recurrent neural networks like LSTM are turing-complete, i.e. for every halting algorithm it is trivial to implement an RNN that computes it. It is non-trivial to learn these parameters from algorithm IO data, but for many tasks it is possible too.

>I suspect that most of what is being learned (and solved) is perception of useful features from the raw game images.

It is not this simple, deep enough convnets can represent computations, the consensus is that middle and upper layers of convnets represent some useful computation steps. Also note that human brain can only do so much computation steps to answer questions when in dialogue, due to time and speed limits.

>My guess is we may need a fundamental breakthrough in a newfangled hierarchical learning system that is better suited for language to “solve” NLP.

This is being worked on, see the first link for Memory Networks and Stack RNNs, DeQue RNNs, Tree RNNs. Deep learning is a very generic term, there are dozens of various feedforward and recurrent architectures that are fully differentiable. The full potential of such models has not been nearly reached yet and maybe language understanding will be solved in the coming years (again, the first link shows that it is in process of being solved).

sapphireblue··on How AI based programming could work
Thank you! I found a great list of deep learning paper summaries that is mostly about learning algorithms from data and other cutting edge models: http://colinraffel.com/wiki/tag:neural_networks?do=showtag&t... all kudos to the owner/author of this list !

>NNs that manipulate existing code? e.g. parse it, find design pattern, optimize

Also: you probably shouldn't need special architectures for such problems, these can be solved by seq2seq, NTM, RL-NTM and such, given a good dataset. Such tasks are still beyond current state of art though, it seems.

sapphireblue··on How AI based programming could work
IMHO talking about AGI here sounds like "all or nothing" type of thinking. Almost all problems can be formulated as [perhaps very hard] machine learning problems of 3 basic types https://en.wikipedia.org/wiki/Machine_learning#Types_of_prob... .

Every ML problem has a lot of algorithms that can be applied to it to solve it. Previously unsolvable problem (e.g. learning playing atari game from pixels) can become solvable if a better ML algorithm is devised, more compute power is applied, or the problem is simplified (maybe with data augmentation or feature engineering, or constraining the domain).

I think the problem you are stating - finding the best configuration of website to extract more ad revenue - can be simplified until it is solvable by currently available algorithms and hardware. Big companies are already doing similar things - Google uses reinforcement learning to recommend new items on its services, Facebook is known to use machine learning to somehow make more profit from users.

Given expert knowledge of the problem and good algorithms, it can be solved in some form, that's my point here.

sapphireblue··on How AI based programming could work
3 years ago many systems that exist today would be regarded as "scifi" or "maybe 10 years into the future". One of the most striking examples is Visual Question Answering with dynamic memory networks [1]. The model is given raw pixels and encoded question in natural language, it should output a natural language answer. See how it works on data it hasn't seen: http://i.imgur.com/074G9pg.jpg

Do not underestimate what good ML model can learn given enough data. And in case of online shopping there usually is more than enough data, and big companies like Amazon have been using ML (context bandits) for quite some time.

1. https://arxiv.org/abs/1603.01417

sapphireblue··on How AI based programming could work
The corpus can be acquired from companies that host coding competitions. In this paper http://arxiv.org/pdf/1510.07211v1.pdf (I already mentioned it many times, but it is very relevant) researchers took the data from a similar source

>To accomplish this goal, we leverage a dataset from a pedagogical programming online judge (OJ) system,2 intended for the undergraduate course, Introduction to Computing. The OJ system comprises different programming problems. Students submit their source code to a specific problem, and the OJ system judges its validity automatically (via running).

And trained seq2seq model on it on character level and it almost works, barring a few typos. IMHO this is an underappreciated breakthrough. With more data and a better model generating competition-grade programs seems possible.

sapphireblue··on MNIST Handwritten Digit Classifier – beginner neural network project
I have an old one written some time ago by myself too, in node.js/javascript: https://github.com/crystalline/dnnjs It is a simple multilayer perceptron with ReLU nonlinearity, it can achieve 1.7% error on MNIST which is bad compared to convnets but good enough for multilayer perceptron. Training a model is as simple as running "node node-mnist.js" in terminal.
sapphireblue··on How AI based programming could work
The author proposes a lot of vague ideas in this article (for example "I believe one of the biggest problems is the use of Error Propagation and Gradient Descent") without references or any solid proofs why they are necessary to solve the proposed program (Automate programming using ML?).

In fact there is already a lot of solid work just on this subject:

* Learning algorithms from examples http://arxiv.org/abs/1511.07275 https://arxiv.org/abs/1410.5401

* Generating source code from natural language description http://arxiv.org/abs/1510.07211

* And, the most closest work to what author probably wants, a way to write a program in forth while leaving some functions as neural blackboxes to be learned from examples: http://arxiv.org/abs/1605.06640

* Also there is a whole research program by nothing less than Facebook AI Research that explicitly aims at creating a conversational AI agent that is able to translate user's natural language orders into programs (asking to the user additional questions if necessary): http://arxiv.org/abs/1511.08130 (there is also a summary here http://colinraffel.com/wiki/a_roadmap_towards_machine_intell... )

And deepmind is also working on conversational agents: https://youtu.be/vQXAsdMa_8A?t=1265

Given current success of such models, automating simple programming tasks maybe not as much research as engineering and scaling up problem.

There is a lot of exciting machine learning research out there nowadays. Almost all of this research is available for free from papers posted on arxiv. It is a really good idea to read more about state of the art before coming with new ideas.

sapphireblue··on How Magic Leap Works – From Field of View to GPU
I find it disappointing that there is this aversive sentiment in tech-scene towards groundbreaking, physical technologies like segways, self-driving cars, and now magic leap. I like Peter Thiel's point of view on this issue: world needs more zero to one and less of the same. Note that I'm talking this as an average software developer (I don't work on cool technology at work). I'd like to live in the future and I acknowledge that without huge risky hardware bets the future will be as bleak as present is.
sapphireblue··on How Magic Leap Works – From Field of View to GPU
With their budget they can afford to make a custom SLAM ASIC, push more cores into their SoC and create some special GPU. Such capital-heavy ASIC approach can give 10-100x performance boost.
sapphireblue··on How Magic Leap Works – From Field of View to GPU
Low-hanging fruit is becoming increasingly more rare. The only real path forward is hard technology. Hard technology requires tolerance for risk, given the necessity of hardtech taking such risks should be celebrated.
sapphireblue··on The future is fewer people writing code?
There are signs, if you know were to look for them: http://arxiv.org/abs/1510.07211 https://arxiv.org/abs/1410.4615 http://arxiv.org/abs/1605.06640 https://arxiv.org/abs/1410.5401
sapphireblue··on The future is fewer people writing code?
Isn't doing programming projects much less cumbersome when you hire a professional programmer and express a high-level project spec to him/her? Why cannot a sufficiently advanced machine-learning model behave just like such a professional?

Modern models learn to achieve goals in increasingly sophisticated 3d environments and even learn execute commands in some very limited form of natural language. You could say that the question of achieving more humanlike performance may be "just" a question of engineering and scale.

sapphireblue··on The future is fewer people writing code?
Seems like a gross overestimation of time needed for said advances and complexity of such systems. There is already a research-grade approach to learn to generate short programming-challenge-like programs from natural language description: http://arxiv.org/abs/1510.07211

>This paper envisions an end-to-end program generation scenario using recurrent neural networks (RNNs): Users can express their intention in natural language; an RNN then automatically generates corresponding code in a characterby-by-character fashion. We demonstrate its feasibility through a case study and empirical analysis. To fully make such technique useful in practice, we also point out several cross-disciplinary challenges, including modeling user intention, providing datasets, improving model architectures, etc. Although much long-term research shall be addressed in this new field, we believe end-to-end program generation would become a reality in future decades, and we are looking forward to its practice.

sapphireblue··on Notes from the fourth RISC-V workshop
That's a good idea, TPU and Nervana show that there is a large potential in such accelerators.
sapphireblue··on Notes from the fourth RISC-V workshop
This is incorrect. With custom ASIC that implements ultra-wide low precision SIMD arithmetic (f16, i16, i8) it is possible to squeeze an order of magnitude or more speedup over conventional GPGPUs using same area and power. GPGPUs have to have big f32 (and even f64!) FPUs, and then some additional 3d-specific hardware overhead, that's why they are still suboptimal for deep machine learning. Google's TPU and Nervana's similar ASIC illustrate this point.

Also you don't need the management cores to be fast, it suffices to have lots of GHz+ cores controlling ultra-wide SIMD (as Intel's Xeon Phi has shown).

Arguably, one could get another order of magnitude speedup by implementing XNOR-net (http://arxiv.org/abs/1603.05279) hardware accelerator, but it hasn't been done yet.

sapphireblue··on Lepton image compression: saving 22% losslessly from images at 15MB/s
Autoencoders are a long known approach to use neural networks for lossy compression. Some papers about using these models as aids for compression are googleable.
sapphireblue··on Lepton image compression: saving 22% losslessly from images at 15MB/s
There is an award-winning compressor that uses many statistical models (and a 3-layered dense neural network) to compress the data losslessly: http://www.byronknoll.com/cmix.html

Also there is a free book that contains descriptions of various common and exotic compression formats http://www.mattmahoney.net/dc/dce.html

Overall I think we are yet to see the full potential of deep learning unleashed on data compression. For example the neural network in cmix compressor is quite primitive compared to modern architectures. Someone will certainly find a way to do better than that!

sapphireblue··on A Proposal For the Dartmouth Summer Research Project on A.I. (1955)
Actually the success of deep learning has made us very optimistic again. Just read this blog post: https://deepmind.com/blog there is a screenshot of environment that deepmind uses to train its agents. It's like science fiction that an algorithm can learn to behave in such rich 3d environment, but it works.

Also see the graphs for atari environments: many games are played by rl agents "at human-level or above".

sapphireblue··on A Proposal For the Dartmouth Summer Research Project on A.I. (1955)
An interesting but not widely known fact is that the Dartmouth conference was also attended by Ray Solomonoff. Contrary to his colleagues he focused on questions of machine learning before learning became perceived as a worthwhile research direction by the AI community.

This has led Solomonoff to investigation of a question of universal sequence prediction. A couple of years later Solomonoff wrote a paper about such a system for prediction that used algorithmic probability (he is cited later as the original developer of algorithmic information theory which was later independently discovered by Kolmogorov who later acknowledged that the Solomonoff was the first). This method, Solomonoff's induction, is proven to be the most optimal (though incomputable) machine learning method possible.

He has never abandoned this project and for the rest of his life he focused on making more sophisticated system designs that are computable while still being proven to be optimal.

His latest system is called "Alpha", and it is designed as a machine for solving a sequence of function inversion and time limited optimization problems (a majority of science/engineering problems can be formulated this way) in a way that exploits experience gathered while solving these problems. This system, again, is proven to be optimal in a certain sense. He also tried to implement this system with various practical optimizations, but it didn't converge fast enough on his training sequences and on the hardware of that time.

Still, with modern hardware it is a possibility that it could work. And the whole design is described in the papers, so people can (and actually do, though privately and perhaps without much success) implement this system.

Here are the relevant papers: http://world.std.com/~rjs/publications/IncLrn89.pdf http://world.std.com/~rjs/nips02.pdf

sapphireblue··on EU regulations on algorithmic decision-making and a “right to explanation”
Explaining how neural network models generate their output is possible, see http://cs231n.github.io/understanding-cnn/ https://auduno.github.io/2016/06/18/peeking-inside-convnets/ https://courses.cs.washington.edu/courses/cse590v/14au/cse59...

Also it should be possible to train the NN model to explain its decision, even in plain language.

I think the "right to explanation" law is pretty sensible, people should have a right to question the algorithmic judgment that impacts their life. It will be somewhat inconvenient for big companies applying ML to make these decisions, but it won't ruin their business.

sapphireblue··on Learn JavaScript and artificial intelligence in a fun, interactive way
I think the grandparent has a valid point. IMHO learning is a necessary feature of any intelligence, artificial or natural. That's why modern AI is 90% Machine Learning. All non-learning approaches (frames, symbolic AI) have failed 20 years ago.

How to make an easily approachable tutorial that combines game-like world and real learning AI (for example Q-learning or maybe evolutionary policy search) remains and open question. Maybe someone will build it.

sapphireblue··on Should we ask governments and corporations to temporarily halt the creation of AI?
1) It would be ridiculously unfair to people that would benefit from AI most: 6 billion of poor people living lives that are unimaginably unpleasant to a western person.

2) Hype about "superintelligent AI coming soon" is pretty much unfounded, current deep learning approaches have a lot of limits, hardware progress stalled as well (moore's law is officially dead). AI technology is like a fragile flower on a life support, it is so weak you can easily kill it with regulation and FUD. Instead of questioning if we suddenly achieve "real AI" one could wonder if the tech will ever reach any level of moderate general AI, considering the death of moore's law.

3) Nobody will listen to scaremongering, people will just continue doing their thing anyway.

← PreviousPage 2 of 3Next →