GradIEEEnt half decent | Hacker News Reader

GradIEEEnt half decent

tom7.org

275 pointsnotmysql_32 comments

Reminds of some earlier openai work https://openai.com/research/nonlinear-computation-in-deep-li...

"Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it’s a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks."

Always delighted to see tom7 content. Dude just thinks out of the box like no one else.

His videos also cover a super wide range of content areas. Just this latest one has a refresher on neural nets, ieee floating point (and explained the accuracy tradeoffs more clearly than I’ve ever seen), fractals, chess engines, and more.

Not only that but they are highly entertaining.

This one is probably my all time favorite: https://youtu.be/LA_DrBwkiJA

Not just that, he is also very good at making videos. His story lines are good and everything flows like a deep river. It's mesmerizing to look at.

and he's also great at naming things, at least based on this one submission

And running the length of every street in Pittsburgh, of all things

It has been a long standing dream to make videos like this. I have no idea where to start though. I think everyone has these out of the box ideas, but we (or at least I) disregard them as "who will be interested in this".

In addition to picking a subject that interests people, it helps to be good at storytelling. Fortunately, that skill can also be learned

Dude is way out of the box. Using IEEE floating point rounding error as his neural network transfer / activation function -- never thought to go there.

Tom7 is kind of like the electronic version of the Primitive Technology Youtuber.

It's fascinating watching someone using some of the worst tools ever to make something in about the most labor-intensive way imaginable - and it's just beautiful. It's practically meditative.

I know what you mean by Primitive Technology Youtuber, but I would encourage you to not dismiss the underlying concepts out of hand. Building a chair from green wood with hand tools may not work for Ikea, but it is much more energy-efficient than power tools [1].

There's also something incredibly empowering about working with tools you understand, both in software and woodworking.

[1]: https://www.mortiseandtenonmag.com/blogs/blog/podcast-12-the...

Oh absolutely. I'm going to confess that I read this book, "Replay" by Ken Grimwood, which is roughly about a guy who keeps having a heart attack on the same day, and wakes up in his younger body, with all of his memories. Again and again and again. Kind of like Groundhog Day, but for an entire adult life. I kind of became obsessed for a while with the idea that I would like to live my life over again. It was a really unhealthy obsession. It took a while for me to live in the moment, try to not hold on to my regrets, etc.

Back to the topic... I kind of watch things like Primitive Technology out of a hope/fear that I'll either get thrown back in time to when these skills were relevant, or that I'll live in a post-apocalyptic world where they're relevant again. (Which was probably inspired by reading Stephen King's "The Stand," and worrying about how my life would be if there was no electricity.)

But yeah, I love watching people show their enthusiasm... I love watching people demonstrate mastery over tools I don't know how to use... I love watching someone build something... Tom7 and Primitive Technology check all those boxes. Adam Savage's Tested often does, too.

> (Which was probably inspired by reading Stephen King's "The Stand," and worrying about how my life would be if there was no electricity.)

Reading ‘A Connecticut Yankee in King Arthur’s Court’ ATA formative age had a similar effect on me. :)

> but it is much more energy-efficient than power tools

Only if you forget to amortize the cost of supporting a human life for the extra marginal hours it takes

Industrialization and automation lead to higher human populations, so if your point was that the cost of supporting a human makes power tools more efficient than hand tools, I think you have it backwards.

Cheap energy makes efficiency less important.

sure, let's all maximise the time we have available to spend on a couch consuming mindless tv.

The "but it takes tiiiiime" argument against making your own things (and achieving a kind of satisfaction, control and learning that's hard to beat) is simply dreadful.

Who are you to decide what people do and don't want to do with their spare time? Some people don't achieve any satisfaction from making their own things; they just see it as a chore. Your disappointment in that is just your subjective opinion, not any reflection of objective truth.

You also seem to think that there's nothing "productive" anyone could do with the time they save from not having to make their own things, which seems obviously false.

It's nice to have the luxury of spare time. Many people don't. Or they have to budget it extremely tightly.

Don't project your use of spare time on everyone else. Some of us use spare time towards productive ends. I would personally prefer not to spend all my time living like a subsistence farmer from 1200 years ago.

Two videos so far this year and we're still in May. What a bliss.

Tom7 our beloved. How might we make this creator publish more videos? Money? Is it money? Can we give him money somehow?

Tom has a pretty good day job, so I suspect that money isn't the thing.

The sigbovik conference seems to be the motivating goal behind these videos. I think the key thing would be to make sure that sigbovik continues and that people are excited about the papers.

Unfortunately, Tom's a lot more creativity and concept-driven than money driven. So the best strategy is probably to spend the money on some kind of near-C sublight vehicle and take a one-hour jaunt in it, then return to Earth in a year and see what's tickled Tom's fancy since you last saw him.

this solution may have side-effects that are left as an exercise for the Rocketeer.

> I think this is a fractal in the sense that it is chaotic, has a color gradient, and could be on the cover of an electronic music album

I didn't know of this criterion

It's on my album "Hausdorff-Besicovitch Criterion"

This is one of the few tom7 videos that I see and am utterly confused by in every way. It feels like he's speaking a different language than I am.

I lucked out a bit in having only recently learned about the need for a nonlinear transfer function (although it was taught to me as a "That's just how we do it," not with the associated underlying explanation that you can't represent all functions with only linear functions).

I believe that this floating point imprecision was something that David Page used (I think it may have been accidental as he was originally doing something else with it?) to achieve a world record training speed on CIFAR10 by summing a bunch of loss values together instead of taking the average. The sum effectively reduced the precision of the loss value and seemed to have a regularizing impact on network training as best as I personally understand. :) <3

Tom7 back to his usual madness. This time, exploiting floating point rounding, first to allow a neural network activation function to be linear (except for rounding errors), then expanding until ultimately making a 6502 emulator and proving that linear operations + rounding errors are Turing complete.

Best start to a video this year, really wish more people would use a hook to get people engaged in learning.

two tom7 videos in as many months, what a treat.

Reply on news.ycombinator.com