Obviously, there are lots of inspiration to be drawn from apps we use everyday, such as GitHub, JIRA, etc, but these behind the scenes view is very informative.
There is material on the internet that are relevant to the topic, but it's quite hard to piece together. After all, there are only a handful of players right now who need to build an API that isn't a REST API.
Among big names, I can think of Zendesk (uses iframes), Coda (runs third-party code on their servers IIRC, isolated via server mechanisms), Salesforce (not sure exactly what they do, but I think they also use Realms as a component to their system).
https://medium.com/zendesk-engineering/sandboxing-javascript... https://trailhead.salesforce.com/en/content/learn/modules/le...
There's a couple of academic papers on JavaScript isolation, but you'll have to do a lot of work to figure out how relevant they are. Be sure to check the publication date.
The folks at Agoric are probably the leading experts actively working on untrusted code isolation in a browser environment right now. I would follow them if you want to hear about the latest new tech: https://github.com/Agoric/SES
https://userdashboard.github.io
I believe this approach was pioneered by Facebook some years ago in the earliest incarnation of their app platforms. There was no iframe sandboxing so they had an intermediate contrived language called "FBML" which compiled to a subset of HTML they allowed. That was the platform where Zynga made their fortune on Farmville originally, just before the iPhone.