I'm in a similar boat, and agree completely that it's super fun to learn this way. Coming from playing guitar, I was already used to starting from chords and listening for the rest, so it wasn't a big change. I'm a big believer in proper timing and solid rhythms being more important than notes, and learning by ear makes those aspects more intuitive, in my experience at least.

One thing I did find was the very helpful, was putting those stickers with note names on each note on the keyboard. They help me keep track of what I'm playing, and when I compose something I rely on the stickers to help me transcribe.