No, it's not about premature optimization but designing towards performance. It's always a balancing act but the earlier you make the right choices the better. There are mandatory questions during the design process, before writing a single line of code. "Is it going to be fast enough?" is one of them and you can't answer it by saying "I'll figure out how to optimize it later" or "computer power will eventually catch up".

Oh, and btw. Stop using the "GPU accelerated" argument as it was a silver bullet. GPU is not a crutch to make slow operations barely usable, it should be used as an extra performance boost for operations that already work.