In-memory sounds like the way to go not just in terms of performance, but in that it makes no sense to build an ASIC or program an FPGA for a model that will most likely be obsolete in a few months at best if you're lucky.
For new foundation models it's even worse, because there's some fancy experiment every time and the llama.cpp team needs two weeks to figure out how to implement it so the model can even run.