In my experience, a lot of performance problems boil down to really stupid problems, like simple code using the wrong data structure out of convenience (e.g. linked lists instead of arrays for lots of randomly-accessed data), or structured in a bad way (e.g. allocating a lot of small pieces of memory all the time). Often times, there are cheap performance wins to be had if you occasionally run the product through a profiler and spend couple of hours fixing the most pressing issue that shows up. Couple of hours isn't much; there's enough slack in the development process to find those hours every month or two, without slowing down your regular work.
But I have a different experience (probably because we work in different areas):
Most the performance problems of the products I ever worked were purely systemic.
They boiled down to technologies and architectures having been chosen for "organizational" rather than technological reasons.
And "organizational" is in quotes because sometimes it was just blackmail: I worked with two developers who quit in protest after the prototype they wrote in Scala was deemed not good enough and dropped for... being too slow, ironically.
In the end the backend was pure and beautiful, but the the frontend devs had to perform joins in the the client and make 21 API calls in a 20-item list and then everything goes to hell.