If you go from 10sec to 1 sec, absolutely.
If you go from 1sec to 100ms, maybe.
If you go from 100ms to 10ms, unlikely.
Assuming "fast enough for a good user experience", expecially early on in a product lifecycle, I prefer to focus on development speed than product speed.
I get your point. This is an outlier, but hear me out:
For some of the workloads I run on https://workers.dev, I can tell you that going from 100ms to 10ms is something I look for every single day. I am also very interested in anything that takes the memory usage down into KBs from high MBs, say.
I'm pretty sure dismissive comments such as yours lead us to this bad place.
Of course by "load" I don't mean TTFB, I mean actually loading the full page, including all the spinners, reflows, banners and toolbars popping into your page, images, etc.