He says "if you want to build a really fast program, pay attention to the performance from the start.". So it's a business requirement from the start (to build a really fast program). That's fine and should be taken into account for development. Put it in your definition of done, start with performance and regression tests or something like that to pay attention to this particular business requirement. Have checks for your performance requirements and handle them as any other requirement for your software.
But most projects simply don't have that requirement and that's fine. Sure, your project will fail if it's unusable and slow but it will also fail if you put your efforts into unneeded optimizations from the start and slow down development of other features. I've seen projects fail due to premature optimizations in the beginning.