Neat, but it doesn't really have any bearing on the deep learning and generative AI trends.
1,081 karma · joined January 5, 2017
Neat, but it doesn't really have any bearing on the deep learning and generative AI trends.
The issue isn't that it can't be done -- in fact, the greatest need right now is for engineers who can come in and build rock solid real world applications on top of commodified neural network architectures and weights, not PhD scientists. Your business might not even use its own ML model! You might just be calling an API.
The challenge is that for a few reasons, it's a very crowded market right now. A lot of people want to make a move into AI, yet for all the hype, the space of viable commercial applications that will survive without indefinite VC funding remains kinda small. Look how AVs are doing after billions and billions in funding chasing one of the most lucrative commercial possibilities imaginable. There's really cool stuff happening industry-wide, and commercial potential is growing, but nowhere near as fast as the cultural hype that has infected certain parts of tech space these past 12 months or so. Plus, many experienced ML engineers and scientists have been dumped back into the job market due to layoffs. So from the hiring side right now, for every AI posting there are tons of applications that have the cool portfolio, and then also a relevant degree and/or prior experience.
That's what you're competing against, so if you're going on portfolio alone it's got to be really outstanding. Way beyond doing the homework for a free course. Learning how to build an ML service that solves an actual problem in the real world reliably enough that you can actually use it should be the goal.
If you happen to be employed at a company where there is a need for an ML engineer in some capacity but no availability (hiring is expensive!), you can try stepping up to help out. Hiring challenges aside, it is absolutely possible to learn on the job the engineering skills needed to, say, build ML infra or work as part of an MLOps team. I recognize that's sort of just up to circumstance though. If you look for a new job, be a little wary of any that want to hire you for a more mundane task (like data entry/cleaning/labeling) with a promise of getting to do the ML engineering stuff, too, "eventually". Such roles do exist but it's also a bait-and-switch tactic.
Anyway, that's what I've got as someone who has been thinking about how to help people looking to do what you're doing, but i hope this thread turns up more ideas too.
Not that there aren't problems to solve regarding AI, just that this line of inquiry won't be relevant to solving them. It'll be complicated boring work dealing with power structures, economics, and social movements, not thought experiments about omnipotent Others.
Are there any good, maybe more gatekept forums for discussing AI news? It is great to have a community to discuss with, the field is so large and moving so fast, and there's so much to learn from other practitioners, but the SNR here (where i often find news) has just gotten so low.
(Though now I'm questioning that too :D)
If the author doesn't like real humans disagreeing with them, they're free to make more chatbots to tell them what a great idea this is.
Choose the sources you trust very carefully, and look to the people actually working on real-world AI systems, not the storytellers and hangers-on.
It's interesting, biological neural networks are known to use inhibitory systems for signal gating and sharpening/sparsification, but it's typically handled by distinct neuron types that are strictly inhibitory.
I wonder if one could boost smaller transformer performance with an analogous construction - add on a small additional module constructed for channel inhibition in the rest of the model.
Right now the main bottleneck for LLM size is GPU memory (VRAM). Training requires much more VRAM than inference, which limits the ability for entities that aren't Google or OpenAI-scale to finetune models (aka do a little more training on your custom dataset).
The paper here suggests that one can actually finetune LLMs with inference-sized VRAM usage instead of training-sized VRAM usage. If true, it will be possible to fine tune larger models on smaller (though still expensive) GPUs -- like a single 3090 instead of 1x or 8xA100s. So, more people can create more customized models.