Interestingly enough, it seems like the KORG wavestate is actually based on some Raspberry Pi, according to KORG's github page, so you can definitely get less than 40ms latency and no stutter. And yeah, Teensy or even FPGA evaluation boards can get you very far, since software development in your spare time is "free". If you search the web, you'll find many examples of such projects. For example I remember some FPGA based project with iirc a small ~50€ Spartan-6 eval board which implemented a fancy 32 voice 6op FM synthesizer.