I started piano when I was 23/24 with the goal of wanting to play Jazz / improvise and quickly realized that the biggest challenge is lacking the technical skills to translate the "mental orchestra" onto the keys.
Especially before bed my mind wanders and produces these amazing Blues/Jazz riffs or Moonlight Sonata-esque ideas, but my talent is far behind my taste when it comes to either trying to play or even transcribing all that I "hear"... at least without significant manual effort.
So yeah, maybe that just confirms what you're saying, but what I realized around year 1 was that "improvisation" is really just a musician's bag of technical skills/tricks combined together in their own unique ways... And ultimately the more technical capability you have, the more "improv" things will appear.