Hacker News Reader

Speculative decoding requires you to load the smaller model into memory and run inference on it.

I think the smaller model is at least 20 times smaller. If you do speculative decoding on a 70B model an 1B model would be appropriate.