I hear you, ideally the compiler should be able to manage any arbitrary input in a reasonable time, and catch itself if necessary. Usually it does—sometimes with complex ungrouped arithmetic operations with ambiguous types, Swift will error out and tell you to break up your expression, rather than get stuck.
I would love for the Swift compiler to be dramatically faster, but I understand the challenge, with a powerful type inference engine that supports Generics. It's a resource scarcity problem. If the Swift team spends 10 hours to handle long strings of integer literals, that's 10 hours they haven't put toward features that would benefit a larger audience.