How far are we from getting a general model that can resynthesize any instrumental audio sound without fiddling with any knobs, so that we can recreate instruments we hear from any song? Seems like it should exist by now?
Like physical modeling synthesis, the interesting part is to compress the sound to some parameters that you can tweak and generate new sounds
Another approach is VAE, which also you give your some latent embedding, you can tweak the embedding to generate new sound. However the meaning of this embedding is not explicit.
This doesn't really work on instruments like guitars. Open D sounds way different than fretted D on the E string. Timbre changes with position and it's one of the ways I determine where a player's hands are on the neck when I'm trying to play their song.
Timbral differences also exist depending on force, the manner plucked, the already ringing overtones... It's hard to know what you want, but the most natural thing is always going to be some organic variation in the notes in general.
If you have a good ear, you aren't, I don't think, hearing so much the timbral diff in the individual open or fretted notes as much as the fact that a barre chord and an open chord is a different voicing of the same harmony.
Before the rise of the steel string and the Spanish guitar, guitars tended to be more even across their range and also had less bass which helped even them out, and now that sound is what we are used to. There have always been niches that wanted that more even sound, but for most that just makes it more difficult to play all that music that developed around these quirks, so they remain niches.
And that's only based on how little I know about this.
Aphex Twins' MIDI Mutant came pretty close (quite a few years ago now) to delivering on that promise:
https://magazine.raspberrypi.com/articles/aphex-twin-midimut...
I'm pretty sure another pass at this problem would prove quite fruitful - as others have noted elsewhere in this thread, there are tools like SynPlant which promise this kind of functionality, although - for my needs - I much prefer AT's approach, having it all in a single box.
Seems to me that the Zynthian/Monome[1] folks might have something like this in their toolkits, somewhere. Might be time to catch up with those projects...