Your statement, 'I don't care how it looks, I just want it to work', versus your argument against it, 'aesthetically pleasing designs and good usability lead to you wanting to use an app more' are perhaps misaligned.
Usability is not supposed to be aesthetically pleasing, it can be but it is not supposed to be. It's supposed to be usable.
If something is more usable by removing a microsecond delay, then it may not look as nice but it is more usable. I would prefer more usable to nicer looking.
Design is absolutely not just vanity, but artificial delays absolutely do hinder usability to increase the vanity. It may not "feel" as nice, but it is objectively better to use.
The Android Chrome browser is a great example. Swiping stuff around, amazing, smooth and great animations. The animations take only as long as it takes for me to use them. But when I click the button to show me more tabs, it takes a moment to swoop out, then I click and it takes a moment to swoop back in. Great, spatial awareness is introduced and it felt nice, neat.
But now I have 5 tabs open and I am trying to flip between them without losing my train of thought and all those delays add up, it is now no longer as nice to use because I feel like I am ahead of my device because it's slow, when it's really just the artificial delays.
You can still move up to the tab overview screen, it still gives spatial context, but the animation is not needed, and can be removed for better usability.
I also feel like good animations can still exist without delay in many contexts (not all). Complete the users action instantly, show them the results instantly, but in some contexts there is room to fade out auxiliary information out more slowly.
Moving up to the tab overview is a good example. The tabs can be shown instantly, and just a ghost of the previous screen could fade out behind it.