A Neural Network in 11 Lines of Python

At the risk of being pedantic...if you're not familiar with data/math work in Python, the `np` word refers to "numpy", which is an extension to Python that includes array and matrix math...so the OP needs 12 lines, the first being:

       import numpy as np

Just for comparison this is almost the same using a library [1]:

  import numpy as np
  X = np.array([ [0,0,1],[0,1,1],[1,0,1],[1,1,1]])
  y = np.array([[0,0,1,1]]).T

  from keras.models import Sequential
  from keras.layers.core import Dense

  model = Sequential([Dense(3, 1, init='uniform', activation='sigmoid')])
  model.compile(loss='mean_absolute_error', optimizer='sgd')
  model.fit(X, y, nb_epoch=10000)
  model.predict(X)

[1] http://keras.io

Thanks for this. It is really well explained. However, it did feel like it went downhill when you explained how the updating happens. At least it was less of a walkthrough than the previous sections.

Also, lines 22 and 23 in the three layer network are not intuitive for a beginner. Before, syn0 was (3,1) and now its (3,4), and the new syn1 is (4,1). Your previous explanation for this initialization was much better.

But again, awesome work, just being nitpicky. Really appreciate it!

Also recommended, 7 part video series (link is video 1):

https://www.youtube.com/watch?v=bxe2T-V8XRs

Great work! I have been reading a practical guide to neural networks [1], and I am amazed that it is so comprehensible. I had assumed ML to be a lot more complicated. Although, I won't classify it as an easy craft but (maybe) it isn't so hard as it looks.

[1]: http://neuralnetworksanddeeplearning.com/chap1.html

Some of you may like seeing the two level neural network code translated to kona (or k3)!

    nonlin:{:[y;x*1-x;1%1+_exp -x]}
    x:(0 0 1;0 1 1;1 0 1;1 1 1)
    y:+,0 0 1 1
    syn0:-0.5+3 1 _draw 0
    step:{L1:nonlin[x _mul syn0;0]
          L1err:y-L1
          L1delta:L1err*nonlin[L1;1]
          syn0+:(+x)_mul L1delta;L1}
    do[10000;L1:step[x;y]]
    L1
  (,0.009668795
   ,0.007862982
   ,0.9935916
   ,0.9921171)

k3 does this in 140ms, Kona in 2030ms on the same machine.

This is a really fantastic article, I've been looking for a code-based "Neural network implementation for idiots" thing like this.

I wonder if it'd work better with a leaky ReLU. Somebody wanna try? It's a _very_ simple mechanism. [1] Basically, instead of a sigmoid, let f(z·w) be the activation due to inputs z and weight w, then a(z) = z·w if z·w > 0 else 0.01 * z·w. Sigmoid for activation have several problems, most notably they suffer from vanishing gradient problems with bigger networks.

[1] https://en.wikipedia.org/wiki/Rectifier_(neural_networks)#Le...

Shouldn't the derivative of sigmod(x) be sigmoid(x)(1-sigmoid(x)) and not x(1-x). The code seems to implement the second.

Can HN recommend any good neural net tutorials? (Preferably in Python)

I done something similar some months ago. Maybe this helps someone though it should have same content: http://nbviewer.ipython.org/github/bodokaiser/data-science/b...

Interesting fact about the sigmoid function: If you scale it so its slope at the origin is the same as the cumulative error function, it differs from that function by at most 0.017.

I don't get the point of these "x in n lines" posts.

Fewer lines of code doesn't always equate to simple, comprehensible code.

The final "Line 36" in your explanation should be "Line 39."

YES! Thank you!

This is amazing stuff. Just want more!

Anyone fancy translating this to C#?

very nice, thank you!