> a job interview with a deep learning startup
I am not aware of any DL startups that are interested in techniques which are incompatible with GPUs and back propagation. I think this is a solo journey through the dark forest for now.
I am not aware of any DL startups that are interested in techniques which are incompatible with GPUs and back propagation. I think this is a solo journey through the dark forest for now.
Enjoy your adventures in the Dark Forest.
I don't know that classical architectures are infeasible to perturb as noted in other comments. I just think we have a really interesting knob to turn in this specific case that doesn't seem to exist in other architectures.
How would you go about constraining the resource consumption (by up to half!) of a classical ANN while preserving realistic hopes that it will recover between training epochs? And, we are talking real resource reduction here not some academic thing. You can see the difference in task manager. That's actually the primary tool I've been using to determine how to tune the parameters of this scheme.
See for example: https://arxiv.org/abs/2404.02258
They have a fixed compute budget which is lower than what the LLM need, and dynamically decide to allocate this compute budget to different part of the network.
So its not exactly what you propose since here the compute budget is fixed (that's the point of the paper: to make the network learn how to allocate the resources by itself) but its dynamic for each part of the network, so it shows that its possible.
In academia there's quite a bit of stuff on neuromorphic computing going on.