Hacker News Reader

Its not supported by TVM yet, but there is support for Qualcomm Hexagon.

You can kinda see some of the supported backends gated behind flags in the cmake file: https://github.com/mlc-ai/relax/blob/mlc/CMakeLists.txt

It might have been more work to convert the model for the RK3588 NPU, even if Rockchip provides an SDK and an automated conversion tool that should help (the SDK includes a simulator for the NPU, so the converted model can be tried on a PC before being deployed on a board like Orange Pi):

https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_npu.html

How many tokens per second do you think we can get out of this 6TFlops NPU?

For prompt ingestion... I dunno.

Unbatched token generation is basically RAM bandwidth limited, as the entire model has to be cycled through for each token. I bet theoretical performance is similar to the GPU, albeit with much lower power consumption.