That's ridiculous and blatantly incorrect. Users may not notice it but that doesn't mean the development team won't find it easier to understand or the support staff won't find it better to troubleshoot. It is possible to write the most convoluted code for any product that functions in the exactly the same way as the the cleanest, most-easy to understand code. The user will not notice the difference but which one would you rather want to be involved in?
Also, there are a lot of code improvements you can do that maybe nobody notices today but will certainly avoid catastrophes in the future. For example, hardening security. Most users won't notice or care about your security policies when you just release your app but once your product starts to gain traction, you should spend time making sure you improve security on a constant basis. Users do not care if you still use MD5 for hashes but you might as well go ahead and rewrite your code to use better hashing algorithm already.
There's also the financial aspect. As the business needs change, maybe you can adapt your code to perform better on cheaper hardware while giving up some potential features that you had planned on. I ran a moderately popular website that could handle 1-2m visitors/day on some beefy hardware but seeing as how most of our traffic came from embeds, I was able to switch hosts and platform and satisfy the critical load using very cheap yet stable hosting. The users did not see any performance or feature change but I saved $700/month.
So yeah, I would definitely improve code if it has more substantial, long-lasting benefits than immediately obvious to end-users.