RTNeural – real-time neural network inferencing engine
github.com
github.com
Generate fully specialized, stand-alone, human-readable C code for AVX-512 neural net inference:
Far outperforms TensorFlow, PyTorch, etc.
Code generator is also stand-alone (no dependencies) and is written in Go.
But, on the NN512 page I don't see how to take an existing trained network (from tensorflow, torch, mxnet, onnx...) to generate the code? I'm not sure whether NN512 does any of the quite efficient layer fusion or layer-op fusions.
How does it compare, with openvino, tvm, tensorrt (I'm restricting myself to C/C++ stuff, but comparing to the python stuff would be nice too)?
When one says 'far outperforms' other libraries I'd rather like to know all that. And ConvNet is quite old now, many state of the art network architecture are going away from pure cnns and going into rnns, transformers, and even deeper networks where I'm not sure AVX512 really helps (except for batching and even then...).
As for 'human readable' that's quite the stretch, if you're not an avx512 intrinsic reader it's quite the sequence of instructions.
Is there a comprehensive, recent, answer to all or some of this in the NN512 docs, that I missed?
Also it seemed last time that I checked that the author had tried to monetize the project somehow and was quite disheartened and was (iirc?) kind of abandoning the effort? I can understand how the ML/AI inference can be discouraging, and how hard it is to follow the big 2 (tf, torch) even for deep pocket orgs. Can't imagine how much people like FINN/Tensil/Tenstorrent will spend supporting all operators... And new ones keep appearing...
Most competent engineers will be able to dump their network weights to a file as described in the headers. Do you understand your network or not? The weights, etc., are literally a C struct in the header file. Use fread() to load the floats. If that's too complicated, NN-512 is not for you.
As clearly stated on its brief webpage (which you might care to read before commenting) NN-512 does every kind of fusion you can imagine, fusing convolutions, fusing elementwise operations, fusing batch normalization, etc. See https://nn-512.com/example/28 for an example.
If you're using a classical ConvNet (e.g., a DenseNet-121), as many people are, and you want to do inference on AVX-512 CPUs, nothing will outperform NN-512. And your inference code will be completely stand-alone. No dependencies.
I didn't mean to be disparaging and I guess that's my fault asking questions or stating doubts on HN but last time I tried, with some not off the shelf models, to use it wasn't 'just' loading structs. There were some ops missing and not every one uses mainstream convnets. I guess I was wrong too and couldn't make some lstms work right aways.
As for the comments on 'competent', not caring to read, asking if I know my networks (as I talked about using other inference libraries that I seem to be able to make work), etc. I think that was uncalled for, but since I seem to annoy you, I promise not chip in on NN512 anymore. Thanks.
All I meant was: it's not hard to write your weights to a file as floats and fread() them into a struct. I view this as the simplest possible universal interface.
And you're right: NN-512 supports only the classical ConvNet operations. It does that one thing, and it does it well. If you need LSTMs or transformers, or training, etc., another tool is needed.
Have a nice day.