OSGi was supposed to solve this problem. You could start by running all bundles together in the same JVM, then split them across a cluster later if required. But it never quite took off beyond an implementation detail of various application servers.
That's an understatement. It was soul destroying awfulness.