These things all feed into each other.
If you're in the (vast, vast) majority of Swift developers then you're writing apps for iOS, MacOS, etc. This means outside of that context Swift goes from being a relatively popular language with a strong ecosystem to an incredibly niche one.
One angle where this could gain traction is devs writing a server side backend for their Apple app - but this use case is sliced apart in practice.
- Teams that start off wanting to use the same language for the app and the backend are likely to pick React Native or similar.
- The larger teams that want/need to write their app natively likely have devs that write the apps and devs that write the server code - so the desire the for language to be the same is lower.
- The pool of developers you could hire that have backend experience and swift experience is much much smaller than either of those two factors alone.
On a pure 'is this language good enough for the problem' level - sure, swift could do the job.
But that's also true of almost every other language.
Take C# for instance: Microsoft has a rich history of being very serious about the enterprise, and was there on the ground floor of the '.com' days with popular server software. MS leveraged knowledge developers had writing Visual Basic with VBS and also Jscript, a JS variant, to popularize ASP, then convinced people to move to C# which let you do both server and desktop with the same knowledge. And all this ran on the Microsoft server OS, a popular product, out of the box.
Let's compare this with Swift. Apple has never, ever been serious about the enterprise, hasn't sold any servers during its whole lifetime, and while I'm sure you can run server side Swift on a real Linux server instead of just a Mac, its relative newness (newer than every popular language but Kotlin) means there would need to be an affirmative reason, a big tangible benefit, to convince anyone to either switch, or to start their whole career/company with Swift without ever learning anything else. Much the opposite in my humble opinion - you have Apple treating developers poorly with their aggressive rent-seeking behavior. I would never want to ditch another language that isn't controlled by one firm, to work on a platform that, though nominally 'OSS,' exists purely for Apple's benefit and is controlled by them.
Server-side Swift has one thing going for it: You can leverage your skills gained making iOS native stuff. Unfortunately, it seems to me that few companies besides indie 'Apple-only' devs even want to use 'iOS Swift' since it's limited to Apple platforms and most companies want cross-platform mobile apps. So the number of people out there who are 'Swift experts' and would find that to be the most compelling server-side environment is utterly dwarfed by people who have that level of mastery of JS, Python, Java, C#, Kotlin, PHP, Ruby, Go, etc. Which is of course a Catch-22, 'nobody uses server-side Swift because it's not popular enough to support a great community.'
To kick off a new project with a Swift backend would be to say "I trust Apple unconditionally, and also I have no intention of ever needing to hire anyone to help with this."
I worked at a place that worked with Delphi, and for various reasons I had to use it exclusively for a few years. No recruiter would touch me. Not until I got some time with Rails did I have a chance to escape.
As a former mobile dev, I'd also like to add, being an app dev vs BE dev isn't just about the code either.. It's a very different way of looking at problems. The skills might transfer, but they're living in different worlds. The language isn't the only obstacle.
Apple has most definitely sold servers during its lifetime. The Xserve line for example.