In fact, I remember looking on my SNES and NES classics at a laundry list of open source libraries used for those mini consoles.
"Free and open source" doesn't mean you have to give things away for zero dollars (free as in beer vs free as in speech), just that you have to provide users the means to inspect, modify, and link what you've released. I think you'll find quite a bit of GPL and open source information in the about section of quite a few games. I seem to recall Darksiders 2 (among others) offering to send you their GPL'd source from the main menu about screen.
So, I think it works out just fine for games. Hell, all the games made from the GPL'd id-tech engines should be enough to convince you it has worked just fine for games.
Think about the business model of, say, charging for support for an open source library. Users don't pay that essentially _ever_. A business will pay, and they'll pay because they produce a non-free product (maybe it's SaaS, maybe they license proprietary software to other entities, whatever).
A game is the end of the line. If it's free software, (essentially) no-one pays for it.
There are also various examples of companies like GitLab that have the nonfree SaaS version and the free (MIT!) community version.
I'd be interested in seeing examples of a pure free software business model.
An example of a pure free software business model is RedHat, they open source everything they make and that is working out for them https://news.ycombinator.com/item?id=18322772
I think that we would see much more of this is proprietary software didn't exist. But of course, re-selling the same development over and over and over again is much more profitable, so it's hard to compete with that.
They all needed software that fit their needs, and none minded that we shared any generic components we built to accomplish it.
If we want to know more, the companies I worked for were members of the https://odoo-community.org/
The biggest potential replacement for Unity seems to be Godot Engine (https://godotengine.org) which is MIT licensed. From what I hear it's supposed to be easier to learn than Unity.