One thing I've wondered in this space: Let's say for a really basic example I want to identify birds and houses. Is it better to make one large model that does both, or two small(er) models that each does one?
An end to end model will always outperform a sequence of models designed to target specific features. You truncate information when you render the data into output space (the model output vector) from feature space (much richer data inside the model), thats the primary reason why to do transfer learning all layers are frozen, the final layer is chopped off, and then the output of the internal layer is sent into the next model. Not the output itself.
Yes you can create a large tree of smaller models, but the performance cieling is still lower.
Please don't tell people to do this. Ive seen millions wasted on this.
When you train a vision model it will already develop a heirarchy of fundamental point, and line detectors in the first few layers. And they will be particularly well chosen for the domain. It happens automatically. No need to manually put them there.
Is there specific tooling to inspect intermediate layers or will they be unintelligible for humans?
The early layers in a vision model are sort of interpetable. They look like lines and dots and scratchy patterns being composited. You can see the exact same features in L1 and L2 biological neural networks in cats, monkeys, mice, etc. As you get deeper into the network the patterns become really abstract. For a human, the best you can do is render a pattern of inputs that maximizes a target internal neurons activation to see what it detects.
You can sort of see what they represent in vision. Dogs, fur, signs, face, happy, sad, etc, but once its a multimodal model and there is time and language involved it gets really difficult. And at that point you might as well just use the damn thing, or just ask it.
In finance, you cant tell what the fuck any of the feature detectors are. Its just very abstract.
As for tooling, a little bit of numpy and pytorch, dump some neurpn weights to a png, there you go. Download a small convnet pretrained network, amd i bet gpt4 can walk you through the process.
Is it feasible for someone with a SWE background with fair amount of industry years to transition into ML without a deep dive into a PhD and publications to show?
I am considering following the fastAI course or perhaps other MOOC courses but I am not sure if any of this would be reasonably taken seriously within the field?
As for getting jobs I cant help you with that part. You'll have to do your own networking, etc.
gibsonmart1i3@gmail.com Shoot me an email if your serious lets schedule a call.
wegfawefgawefg - I bookmarked this and worked through it more carefully when I had time, I appreciated the learnings.
As for your ammendment, it is good to reduce compute when you can, and reduce up front effort for model creation when you can. Reusing models may be valid, but even in your ammended process you will still end up not reaching the peak performance of a single end to end model trained on the right data. Composite models are simply worse, even when transfer learning is done correctly.
As for the compute cost, if you train an end to end model and then minify it to the same size as the sum of your composite models it will have identical inference cost, but higher peak accuracy.
You could even do that with the "Shared Backbone" architecture, as youve described where two tailnetworks share a head network. It has been attempted thoroughly in the Deep Reinforcement Learning subdomain I am most familiar, and result in unnecessary performance loss. So it's not generally done anymore.
Just that its peak performance is lower than an end to end model, and that if youre going to encourage model kit-bashing be clear how you communicate it, so people dont make human centipede architectures and wonder why feces is what comes out the end.
I was a very polite enough "some guy on the internet". Thank you.