Hacker News Reader

Comment by wpietri | Hacker News Reader

Have folks seen good Linux tools for actually monitoring/profiling the GPU's inner workings? Soon I'll need to scale running ML models. For CPUs, I have a whole bag of tricks for examining and monitoring performance. But for GPUs, I feel like a caveman.

View on HN

For NVIDIA GPUs, Nsight systems is wildly detailed and has both GUI and CLI options: https://developer.nvidia.com/nsight-systems

For DL specifically, this article covers a couple of options that actually plug into the framework: https://developer.nvidia.com/blog/profiling-and-optimizing-d...

nvidia-smi is the core tool most folks use for quick "top"-like output, but there is also an htop equivalent: https://github.com/shunk031/nvhtop

A lot of other tools are build on top of the low-level NVML library (https://developer.nvidia.com/nvidia-management-library-nvml). There are also Python NVML bindings if you need to write your own monitoring tools.

I personally like gpustat -- it's a nvidia-smi wrapper but it has colors...

They also i guess now have a web sever plugged into it which seems pretty cool

https://github.com/wookayin/gpustat https://github.com/wookayin/gpustat-web

In nvidia land, `nvidia-smi` is like `top` for your gpus. If you're running compiled CUDA, `nvprof` is very useful. But I'm not sure how much work it would take to profile something like a pytorch model.

https://developer.nvidia.com/blog/cuda-pro-tip-nvprof-your-h...

In AMD land, `rocm-smi` is like `top` for your gpus. Very similar in capabilities and functionality to NVIDIA.

https://developer.nvidia.com/nsight-compute

Reply on news.ycombinator.com