If you're hitting a memory wall it means you're not scaling. This stuff really doesn't apply to scaled up inference but rather local small batch execution