20x Faster Background Removal in the Browser Using ONNX Runtime with WebGPU
img.ly
img.ly
But here’s something I don’t understand: (And someone please correct me if I’m wrong!) - now I do understand that NNs are to software what FPGAs are to hardware, and the ability to pick any node and mess with it (delete, clone, more connections, less connections, link weights, swap-out the activation functions, etc) means they’re perfect for evolutionary-algorithms that mutate, spawn, and cull these NNs until they solve some problem (e.g. playing Super Mario on a NES (props to Tom7) or in this case, photo background segmentation.
…now, assuming the analogy to FPGAs still holds, with NNs being an incredibly inefficient way to encode and execute steps in a data-processing pipeline (but very efficient at evolving that pipeline) - doesn’t it then mean that whatever process is encoded in the NN, it should both be possible to represent in some more efficient representation (I.e. computer program code, even if it’s highly parallelised) and that “compiling” it down is essential for performance? And if so, then why are models/systems like this being kept in NN form?
(I look forward to revisiting this post a decade from now and musing at my current misconceptions)
1. They can produce approximate solutions which are often good enough in practice and faster than exact algorithmic solutions.
2. Neural networks benefit from billions of dollars of research into how to make them run faster, so even if they technically require more TFLOPs to compute, they are still faster than traditional algorithms that are not extremely well optimized.
Lastly, development time is also important. It is much easier to train a neural network on some large dataset than to come up with an algorithm that works for all kinds of edge cases. To be fair, neural networks might fail catastrophically when they encounter data that they have not been trained on, but maybe it is possible to collect more training data for this specific case.
I have not discussed any methods to compress and simplify already trained models here (model distillation, quantization, pruning, low-rank approximation, and probably many more that I've forgotten), but they all tip the scales in favor of neural networks.
It's an old quote that, although not 100% accurate anymore, still sums up my feelings quite nicely.
https://towardsdatascience.com/neural-networks-as-decision-t...
https://arxiv.org/abs/2210.05189
I haven't reviewed any of it, I only know of it tangentially.
https://www.semanticscholar.org/paper/Converting-A-Trained-N...
Distilling a Neural Network Into a Soft Decision Tree https://arxiv.org/abs/1711.09784
GradTree: Learning Axis-Aligned Decision Trees with Gradient Descent https://arxiv.org/abs/2305.03515
It occurred to me that NNs ("AI") are indeed a bit like crypto, in the sense that both attempt to substitute compute for some human quality. Proof of Work and associated ideas try to substitute compute for trust[0]. Solving problems by feeding tons of data into a DNN is substituting compute for understanding. Specifically, for our understanding of the problem being solved.
It's neat we can just throw compute at a problem to solve it well, but we then end up with a magic black box that's even less comprehensible than the problem at hand.
It also occurs to me that stochastic gradient descent is better than evolutionary programming because it's to evolution what closed-form analytical solutions are to running a simulation of interacting bodies - if you can get away with a formula that gives you what the simulation is trying to approximate, you're better off with the formula. So in this sense, perhaps it's worth to try harder to take a step back and reverse-engineer the problems solved by DNNs, try to gain that more theoretical understanding, because as fun as brute-forcing a solution is, analytical solutions are better.
--
[0] - Which I consider bad for reasons discussed many time before; it's not where I want to go with this comment.
Not if NNs are complex systems[1] whose useful behavior is emergent[2] and therefore non-reductive[3]. In fact, my belief is that if NNs and therefore also LLMs aren't these things, they can never be the basis for true AI.[4]
---
[1] https://en.wikipedia.org/wiki/Complex_system
[2] https://en.wikipedia.org/wiki/Emergence
[3] https://en.wikipedia.org/wiki/Reductionism, https://www.encyclopedia.com/humanities/encyclopedias-almana..., https://academic.oup.com/edited-volume/34519/chapter-abstrac...
[4] Though being these things doesn't guarantee that they can be the basis for true AI either. It's a minimum requirement.
import AppKit
import VisionKit
@main
struct Script {
static func main() async {
let image = NSImage(contentsOfFile: "input.heic")!
let view = ImageAnalysisOverlayView()
let analyzer = ImageAnalyzer()
let configuration = ImageAnalyzer.Configuration(.visualLookUp)
let analysis = try! await analyzer.analyze(image, orientation: .up, configuration: configuration)
view.analysis = analysis
let subjects = await view.subjects
for (index, subject) in subjects.enumerated() {
let subjectImage = try! await subject.image
let pngData = NSBitmapImageRep(data: subjectImage.tiffRepresentation!)!.representation(
using: .png, properties: [:])
try! pngData?.write(to: URL(fileURLWithPath: "subject-\(index).png"))
print("subject-\(index).png")
}
}
}I only skimmed the article, but I don't think they mention the size of the image. 100ms is not that impressive when you consider that you need to be three times as fast for acceptable video frame rate.
You don't need three times as fast for acceptable video frame rates in a video editor, you need a system that allows you to cache "rendered" frames so when the user does an edit, it renders to this cache, then once done, the user can play it back in real-time.
This is essentially how all video editors handle edits on clips/video today. Some effects/edits can be applied in real-time, but the more advanced one (I'd say background removal being one of them) works with this type of caching system.
WebGPU is at least one year away of becoming usable for cross browser deployment.
In Firefox it seems to be behind a feature flag and Safari seems to have it in it's "Technology Preview" (some sort of release candidate?), so seems closer that I at least though.
WebGL 2.0 took almost a decade to be fully supported, and still has issues on Safari, don't expect WebGPU to be any faster.
Also note that Google is the culprit why WebGL Compute did not happen, WebGPU was going to sort all problems, and even though they use DirectX on Windows, apparently it was a big issue to use Metal Compute on Apple instead of OpenGL, and then they ended up improving Angle on top of Metal anyway.
Web politics.
You can combine both today alreay and I experiment with it. The problem is still the high latency of the GPU. It takes ages, to get an answer and the timing is not consistent. That makes all scheduling for me a nightmare, when dividing jobs between the CPU and GPU. It would probably require a new hardware architectur, to make use of that in a sane way, so that GPU and CPU are more closely connected. (there are some designs aiming for this, as far as I know)
edit: you probably meant hamsterS.js
Although the segmentation quality is much better than that of `rembg`, the interface to it is just unamazing. Update: nope, it's sharper, but fails at different images at the same rate.
gist: https://gist.github.com/sou-long/5c7cfee57f5399918c9072552af... (adapted from a real project, just for reference)
Or do they do it server side?