The Little Book of Deep Learning [pdf] | Hacker News Reader

The Little Book of Deep Learning [pdf]

fleuret.org

521 pointsdataminer36 comments

Usually I'm a harsh critic of text like this because casual language and what look like casual words but are actually strictly defined domain specific technical definitions are utterly indistinguishable.

However! This book back-references its appendix with grey underlines thus signaling that something has a technical definition and is jargon.

For instance, "loss" is really common English. In ML it's a loss function. You can clearly see it's a special word in this world in the text. Other examples include weights, capacity and channel.

I don't have to sit there confused trying to guess which English words are being used in special ways.

This discipline is fantastic. In some texts, such terms might be italicized but that behavior has seem to fallen out of practice.

As someone who isn't a professional mathematician, hints like these help greatly.

I fully agree with your general critique. It is a common bad habit in science to use common words with an uncommon meaning without any explanation. I remember me sitting in a lecture about cryptology and wondering how that "I can prove to know a SECRET without revealing it" is supposed to work. I just did not realize that SECRET just meant "random-looking string" instead of something like "I know where the money is hidden".

If it's a map to the hidden treasure (as in the exact bits of the image of the map) then it's kind of the same thing right? But yes, the non-uniqueness of expressing similar ideas can make things difficult there.

You say this about exactness, but my experience with much of science and ML in particular is the opposite.

People still haven't decided what the difference between AI and ML is, for instance, but somehow still insist to treat the two as obviously separate in conversations.

My experience has been that the naming problem is alive and well in the sciences; and it'sfar more problematic than in programming and variable naming.

> AI and ML ... still insist to treat the two as obviously separate

Automated problem solving does not imply "learning". E.g. clustering does not. Also, expert systems are pretty static, and it is not really the "machine" that learns (a flowchart can hardly be said to "learn").

> decided

It is not a matter of deciding ("de-cidere" (cutting) and fuzziness do not really marry), it is just looking at the terms with some historic awareness.

The "brain" works through fuzzy patterns, and of course the concepts somehow relevant to said model also do. You don't cut over a blurry line, but separate points in space remain distant.

The way you present the distinction is really clear, but in the physical sciences it’s very common to use the terms AI and ML interchangeably, and I think it’s partly because people in those fields are not quite sure of the distinction between the two terms

It’s ML when you’re doing research and AI when you ask for funding ;).

>People still haven't decided what the difference between AI and ML is, for instance, but somehow still insist to treat the two as obviously separate in conversations.

Strange, in most of the papers and the books about this they define AI and ML and their relation.

Yes, but the definitions tend not to be consistent.

The best operational definition I got to was, it's ML when it involves a "machine" (possibly defined in software) that tries to 'optimise' some objective on behalf of a user; whereas AI involves some sort of Agent, who needs to demonstrate intelligence, that involve particular environments / states contexts, etc.

But, not everyone defines it like this. To many people AI is rapidly becoming "Neural Networks, particularly Deep ones", with ML becoming "Anything that is in scikit learn that isn't a neural network". Which isn't really a useful definition if you ask me.

Kilian Weinberger's 1st lecture on YouTube.

1956 (or 57) the perceptron was made. The public mood was ecstatic. Generals were dreaming of robot armies, finance world was dreaming of automation, and the whole world was dreaming.

1964 (or 66?) a bestselling book was AI Winter. It explained in everyday English how the whole thing was a farce (at that time). Funding dried up. It wasn't sexy to say AI anymore. So people started saying ML. For funding they were doing ML and not AI.

Nowadays it is a top down vs bottom up approach. One is AI and the other is ML (i don't know which, it doesn't matter)

I had a course named `Deep Learning` by François Fleure at EPFL, and he is a really an amazing professor, able to share his passion alongside explaining very advanced topics. It does not surprise me to see that his books are of very high quality !

The course was focusing on all of the mathematical aspects of Deep Learning, starting from the simple understanding of the gradient descent algorithm to (trying to) understand how transformers work. There was also quite a lot of computer science involved and lots of practical assignments. One of the 2 projects of this course was to design from scratch in Python or C++ a DNN framework (roughly an API like pytorch or tensorflow) which required to really think properly about which architecture to use for your code. The minimum requirements only asked to implements a few activation, normal layers and convolutional layers but you go beyond that and implements all kind of layers. Lots of fun. This course remains as one of my favorite courses.

He runs a very similar course at the University of Geneva (the practicals are available). https://fleuret.org/dlc/

> practicals are available

Also the videos (slides + lesson). Partytime! ;)

Thanks for sharing. This looks great!

By "little" I thought it would be brief, which I think it sort of is. But what it really means by little is that it is optimized for little screens such as a smartphone.

Go ahead and open it in your phone. You'll be delighted to read it.

Question for HN: how can I convert my existing PDFs and eBooks that I can easily read from my phone?

For e.g., I have a lot of Math textbooks in PDF format and I would like to convert them into a format similar to this deep learning book. How can I go about doing that?

Reads great on Kindle too!

To answer your question: Calibre might be able to help you out!

Look into k2pdfopt. The term you’re looking for is you want to “reflow” your PDFs

https://github.com/koreader/koreader

koreader has a pdf reflow mode

Your comment prompted me to look around. There is also an A5 version of the book

https://fleuret.org/public/lbdl-a5-booklet.pdf

Which again, is a small paper size!

Good find! Though the inversion of every other page makes it difficult to read unless printing out

His website also has a version for printing out on 36 sheets of paper and folding into a real book:

https://fleuret.org/public/lbdl-a5-booklet.pdf

What a neat little book! I suppose this was typeset in TeX. How did they optimize for smaller width though?

Edit: I found the style templates on author's webpage [1].

[1]: https://fleuret.org/cgi-bin/gitweb/gitweb.cgi?p=littlebook.g...

If you have texlive-extra installed, this mobile-friendly PDF can be converted to a desktop-readable format using:

  pdfxup -o lbdla4.pdf -ow -im 10 -m 40 -is 0 -fw 0 --portrait -nup 2x2 lbdl.pdf

Are there arguments to that to tile it by columns first instead of rows? (but not one long column all the way down the left to the end and then one long column on the right, just want to shuffle the order on each page).

It reads kind of r/nosafetysmokingfirst to me.

(And its `brew install texlive` on mac to get pdfxup for anyone else wondering)

I know La/TeX is the standard for this but I'm still irritated that it forces all diagrams to appear at the top of the page, which really gets in the way of explaining things clearly.

This books suffers a lot from this limitation, with figures sometimes appearing 5 pages before they are actually referenced in the text.

I am not sure if that is entirely correct. For e.g., see p.no. 14-15 where the diagrams appear at the bottom of the page.

But, I agree that usually it is not easy to place images where you want in LaTeX.

Great writing! I love how this book gets to the point right from the start, answering one of my questions about deep learning (how did this start?) in the very first sentence:

"The current period of progress in artificial intelligence was triggered when Krizhevsky et al.[2012] showed that an artificial neural network (..) could beat complex state-of-the-art image recognition methods by a huge margin (..)"

This is such a amazing little book. I love it, AI is hard for me to understand but this book makes it very easy to grasp some of the concepts.

Thanks for sharing

Andrew Ng's courses are great too btw

> AI is hard for me to understand

Periodic reminder that the lessons (of classical AI - the base) of the late Prof. Patrick Winston, MIT, are freely available (at MIT OpenCourseware, also on YT).

Any recommendations for a set of hands on exercises that go well with this book ?

Probably his course, all in pytorch.

https://fleuret.org/dlc/

For my level of knowledge, this is a bit advanced book. Anything more basic?

You haven't specified in which way this is too advanced for you.

Still, I'll take a stab at recommending some other resources.

1: Practical Deep Learning by fast.ai - a (free) hands on course that's designed to get you to making something useful as quickly as possible.

2: Neural Networks Zero to Hero from Andrej Karpathy - A Series of Youtube Lectures that requires only basic python knowledge and takes you all the way up to a simplified implementation of the tech inside gpt models.

3: Neural Networks from Scratch by Sentdex. A book that's explicitly written to teach you the basics. Doesn't cover stuff like Transformers and other advanced concepts, but really takes its time with the basics.

Take a look at all three of them. They all have a different teaching style and I'd guess that at least one of them will gel with you.

This helps. Thanks.

Careful not to swallow this.

Reply on news.ycombinator.com