Compiling is CPU-bound and doesn't really bottleneck on anything that libc is involved with other than the allocator, so unless musl's allocator is comically slower than glib's, chances are it's just -Os.
This release features the new "mallocng" malloc implementation, replacing musl's original dlmalloc-like allocator that suffered from fundamental design problems.
https://musl.libc.org/releases.html Rest of the detail in the actual changelog is worth reviewing too