You can outright disable transcoding. That’s a checkbox in the server option.
> both this and the native Android app do not handle Dolby vision or passthrough atmos content properly
They do out of the box. Never had an issue with either transcoding disabled. If you have properly configured the source content, Jellyfin is just a passthrough for Kodi and content is directly accessed using the network share.