Hacker News Reader

Comment by phkahler | Hacker News Reader

Does it work with an APU? I just put 64GB in my system and gonna drop in a 5700G. Will that be enough? SFF inference if so.

View on HN

I'm able to run Ollama and llama.cpp on my Ryzen 4600G APU following this guide: https://agieverywhere.com/apuguide/AMDAPU/APU_Linux

Your APU should be similar, just faster.

There are some magic environment variables you want to set to get ROCM to work with this technically unsupported APU: HSA_OVERRIDE_GFX_VERSION=9.0.0 HSA_ENABLE_SDMA=0

Performance is not great, but slightly better than running inference on the CPU, with the bonus that your CPU is essentially free for other tasks even while running LLMs.

The integrated GPU of the 5700G uses old architecture from 2017, this one: https://en.wikipedia.org/wiki/Radeon_RX_Vega_series Pretty sure it does not support ROCm.

BTW if you just want to play with a local LLM, you can try my old port of Mistral: https://github.com/Const-me/Cgml/tree/master/Mistral/Mistral... Unlike CUDA or ROCm my port is based on Direct3D 11 GPU API, runs on all GPUs regardless of the brand.

@Const-me according to this it should work, https://github.com/ROCm/ROCm/issues/2216

haven't tested it, but it should according to https://github.com/ROCm/ROCm/issues/2216

You just need to update the version check here

https://github.com/slashml/amd_inference/blob/4b9ec069c4b2ac...

feel free to open an issue, with the requirements and we will test it.

Reply on news.ycombinator.com