Hacker News Reader

Comment by hn_throwaway_99 | Hacker News Reader

Somewhat off topic. As someone who did some neural network programming in Matlab a couple decades ago, I always feel a bit dismayed that I'm able to understand so little about modern AI given the explosion in advances in the field starting in about the late 00s or so with things like convolutional neural networks and deep learning, transformers, large language models, etc.

Can anyone recommend some great courses or other online resources for getting up to speed on the state-of-the-art with respect to AI? Not really so much looking for an "ELI5" but more of a "you have a strong programming and very-old-school AI background, here are the steps/processes you need to know to understand modern tools".

Edit: thanks for all the great replies, super helpful!

View on HN

A course by Andrej Karpathy on building neural networks, from scratch, in code. We start with the basics of backpropagation and build up to modern deep neural networks, like GPT.

https://karpathy.ai/zero-to-hero.html

For a while now, an answer I've seen is to start with "Attention Is All You Need", the original Transformers paper. It's still pretty good, but over the past year I've led a few working sessions on grokking transformer computational fundamentals and they've turned up some helpful later additions that simplify and clarify what's going on.

You can quickly get overwhelmed by the million good resources out there so I'll keep it to these three. If you have a strong CS background, they'll take you a long way:

(1) Transformers from Scratch: https://peterbloem.nl/blog/transformers

(2) Attention Is All You Need: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547de...

(3) Formal Algorithms for Transformers: https://arxiv.org/abs/2207.09238

Part of the problem with self studying this stuff is that it's hard to know which resources are good, without already being at least conversant with the material already.

That problem doesn't really disappear with teachers and classes ;)

YouTube channel that explains the paper in detail: https://youtu.be/iDulhoQ2pro

And subsequent follow-ups (ROME, editing transformer arch): https://youtu.be/_NMQyOu2HTo

I find the channel amazing explaining super complex topics in simple enough terms for people who have some background in AI.

Yannic Kilcher is great but this video worked better for me:

"LSTM is dead. Long live transformers!" (Leo Dirac): https://www.youtube.com/watch?v=S27pHKBEp30

I second the recommendation for Peter Bloem’s tutorial.

I’m also about to read the transformer chapter from this excellent upcoming book by Simon Prince:

udlbook https://udlbook.github.io/udlbook/

The obvious answer is Fast AI's Practical Deep Learning for Coders - https://course.fast.ai/

They'll also be releasing the "From Deep Learning Foundations to Stable Diffusion" course soon, which is basically Part 2 of the course.

I think the concepts are simple. After all, everything is just a mutli-variable derivative. However, I find the choice of notation very confusing. Mostly because it's impossible to remember the shape of everything.

Even if this linked post, they have a "notations" section at the top. Almost immediately, they start using a value k that isn't defined anywhere.

Not that I think statistics terminology or notation is worthy of praise (it’s mostly horrible), but it frustrates me to no end how the ML world reappropriated terms seemingly just to be different.

I think this book is excellent explanation of some architectures, how they work, and how you can quickly build some with Keras

https://www.manning.com/books/deep-learning-with-python-seco...

Written by François Chollet, the creator of Keras

I put together a repository at the end of last year to walk through a basic use of a single layer Transformer: detect whether "a" and "b" are in a sequence of characters. Everything is reproducible, so hopefully helpful at getting used to some of the tooling too!

https://github.com/rstebbing/workshop/tree/main/experiments/...

For something totally different, try “Geometric Deep Learning”. There is a course by the authors of the book.

Once you understand how everything is operations on graphs, most kernels and functions become very easy to understand.

http://cs231n.stanford.edu/ is good for convolutional networks.

Reply on news.ycombinator.com