Non-text file editing is done in a 3d model or image editing app.
There are the rare engines with no editor to speak of -- where things are either done programmatically or other textual definitions -- but they're very very very few and far between.
The engine itself doesn't have a UI, but working with any major engine without using their editor is functionally impossible.
3D assets can be made in your 3d program of choice.
Skeletons and animations are also somewhat standard. Use whatever tool you prefer.
Textures are bound to 3d assets, and use plenty of 3rd party tools.
USD makes scenes interchangeable. Use or write tools to fit your needs.
compute shaders are also somewhat standard. Work them out in python until you're happy with them.
Thats alot of it, if you really hate all game engine UIs. But the UI was good enough for the developer of the engine so mabie it's good enough as is.