A lot of that is likely because of the way most REST APIs are laid out.
Suppose you're looking at a page that displays an invoice for an accounting system. With a traditional non-SPA app, you'd do a bunch of SQL queries and build a page from an HTML template.
With a REST API, you're doing one HTTP call per entity, at least, and then subsequent HTTP calls for aggregate entities.
When I've made SPAs before, I've often eschewed REST and just made a single Ajax call for "everything I need to display X to the user," not just out of efficiency, but also out of laziness -- it's easy to build. As long as you internally use your API only and don't publish keys for third parties, it's easy to maintain too.
There are some solutions to address the whole problem on a grander scale (GraphQL comes to mind), but overall, that's the cause of slow SPAs in my experience.