Hacker News Reader

antinucleonon Nvidia AVO scores 100% on the ARC-AGI-3 interactive reasoning benchmark

AVO’s paper author (ex-NVIDIAN) is here. This work was done half a year ago for GPU kernels, and the same approach has now been applied to ARC-AGI-3. I think people are still underestimating the evolution progress; e.g., recently we made a self-improving evolution harness that generated an entire inference stack and is better than SGLang/vLLM on various tasks: https://int21.ai/insights/addressing-the-inference-bottlenec...

antinucleonon Render a neural network into CUDA/HIP code

Yes

antinucleonon Render a neural network into CUDA/HIP code

Mojo is trying to create a new language to solve the problem, and specialized for CPU. We are using a more pragmatic way to solve GPU AI computation problem.

antinucleonon Render a neural network into CUDA/HIP code

We haven't compared yet.

antinucleonon Render a neural network into CUDA/HIP code

We developed AITemplate majorly for Meta's focus at that time, eg Ads/Ranking need. For HippoML is startup we are building for Generative AI. HippoML is not using AITemplate.

antinucleonon Render a neural network into CUDA/HIP code

It is actually non-trivial to get GPU run fast, especially on SoC with strong CPU like M2.

antinucleonon Render a neural network into CUDA/HIP code

Hippo is faster than AITemplate, and supports more generative models. We haven't compared vs TVM, but for absolute token/s on M2 Max, Hippo is able to run decoding on LLAMA with datacenter level GPUs performance (with other SW).

antinucleonon Render a neural network into CUDA/HIP code

We will disclose more details very soon.

antinucleonon Render a neural network into CUDA/HIP code

Yes. We support >= 1bit <= 16bit models out of box for various of models.

antinucleonon Render a neural network into CUDA/HIP code

AITemplate's original designer is here. We quit Meta in January and start HippoML (https://hippoml.com/). We just disclosed our new engine's performance on LLM: https://blog.hippoml.com/large-language-model-inference-from... On Apple M2 Max our new engine encode/decode is 13.8X/2.4X faster than llama.cpp

antinucleonon Learning to Optimize Tensor Programs

CuDNN v7 was used in the experiments, in the experiments parts each comparison was listed with version or commit number.

antinucleonon Learning to Optimize Tensor Programs

Summary: Tensor program is able to be optimized by using machine learning and transfer learning. The numerical program optimization model is trained on feature from low-level AST of the program. Experiments: Tasks: ResNet, MobileNet, LSTM LM, DQN Hardware: CUDA/ARM GPU/ARM CPU Speed up compare to CUDNN, TensorFlow Lite and ARMComputeLib: ~from 1.2X to 3.8X faster in end-to-end test.

antinucleonon MXNet – Deep Learning Framework of Choice at AWS

The version tested in paper should not have P2P,so it was much slower than current version.

antinucleonon MXNet – Deep Learning Framework of Choice at AWS

There is a huge distributed performance advantages vs TensorFlow. You can get a hint from Prof. Carlos Guestrin's keynote talk at Data Science Summit 2016. Also, CMU CS Dean Andrew Moore cited MXNet as "is the most scalable framework for deep learning I have seen"

antinucleonon Deep Learning in a Single File for Smart Devices

The author is here :) Thanks for your suggestion. The detail of network modification will be released in my thesis, there is a "fast good net" but currently I don't have time to make it faster, so I simply use fast poor net.