FFT is almost never not worth it. Even at small scales, you should be using the FFT (and you are a daily user in audio/video/image contexts).
I've benchmarked FFT/non-FFT based approaches for almost everything in that scale for the last ten years and gotten quite different results.