GH hosted runners use shared hardware so the performance is never good. There are quite a few options available. Harness CI offers hyper optimized build infrastructure, paired with software intelligence (caching, running subset of tests based on the code change) can reduce build times up-to 4X compared to GH Actions.

Are you hosting your own runners or relying on GitHub's?

Blacksmith is your buddy. Its free and just has better images for single-core operations. Unless you're Google, I can guarantee it's faster.