I have to admit that I didn't dive 100% into what made this difference exactly. It just worked better without webworkers.

I just dived into the git history, and it turns out I didn't use the buffers as transferable. Perhaps that was it! I'll definitely check that out later, thank you for pointing this out!