Hacker News Reader

Comment by sillysaurusx | Hacker News Reader

If anyone's reading over this and feels "Gosh, I'll never be an ML dev; this is way too much": I don't know most of that list, and still manage to be a productive researcher. I learn what I need as I go.

That's probably the optimal strategy. I'm skeptical of first-principles learning. It's great to immerse yourself in theory, but when you've gone all the way to "topology" you've probably gone beyond the limit of what most ML devs care about on a day-to-day basis.

It's still useful to know. I've applied lots of ideas from other fields. But can you force that knowledge by forcing yourself to study other fields? Maybe. We all have a finite amount of time though.

That said, lots of items on this list are key, and it'd be worth ranking them. There's no need to memorize formulas for Adam, but knowing the concept of momentum-per-weight is pretty crucial.

View on HN

The author and you seem to be talking to different audiences, you're talking about ML eng and OP is talking about ML researchers.

Researchers absolutely need to know a lot, not necessarily all the way to topology or w/e but definitely the underlying mathematical principles in order to advance the field (IMO).

You're right, I wasn't clear. But I'm a full-time ML researcher. In terms of advancing the field, my contributions so far have been modest, but they're there. Some of my favorite ideas were swarm training (https://battle.shawwn.com/swarm-training-v01a.pdf), stop loss for stabilizing GAN training (https://twitter.com/search?q=from%3Atheshawwn%20stop%20loss&...), and getting GPT to play chess (https://www.theregister.com/2020/01/10/gpt2_chess/) back when that was a shocking idea. And in terms of cited research, https://arxiv.org/abs/2101.00027 has been the most impactful.

There's significant overlap between ML research and ML dev. If I can do it without most of that list, it should give people here some hope of joining the field without needing to immerse themselves in theory.

Aha! Nice, I'm actually booking up on transformers as part of my doctoral programme and had a recollection about your gpt chess project earlier today!

I'm an ML researcher, and I'm with sillysaurusx on this. I actually know most of the things on the list, but only because my research is mainly in model compression and computational efficiency. Recently I've been interested in adapting diffusion models to generate music (in raw audio domain), and I'd say only 5 out of 18 bullet points in ML section are relevant - the rest falls from "nice to know" to "irrelevant".

> Gosh, I'll never be an ML dev

FWIW very few of the ML devs I work with have PhDs. I'm not sure aspiring ML devs are the intended audience.

I feel like I know up to 75% of that stuff and can't get hired even for entry level data science.

Same. Have the PhD also. Never even go to first round of interviews, or hear back.

And to get to this level of understanding takes around a decade. Feels like a huge waste of time. The things you could be in a decade of work...

In my field we’d never hire someone with a PhD as entry level, we’d move right past them. It’s maybe because none of us have PhDs and so we think of you guys as gods that should be above us and we don’t deserve you and we can’t pay you enough. Maybe the same thing is happening?

I don't have a phd degree and I'm not at the same level as someone who has that degree, actually. Just knowing the things that are listed in that post doesn't get you a degree. I'm also circumspect on what I mention in an application for that reason.

Oh sorry I misunderstood your post in the mental context I had at the time.

Reply on news.ycombinator.com