My own anecdotal example. I stopped playing music years ago because I thought I wasn't good, despite liking it. I started playing again because I had no games and I noticed that I should have gotten back to it sooner, despite not being music material. Not only I was able to play stuff that I liked, I learned stuff that I thoroughly enjoyed. Some music were really hard at first, but as I progressed (very) slowly through weeks, I could see my skills improving and I was very satisfied to watch it (slow as it was).
If you like it, I think you should do it, despite of others. Because it won't matter, in the end. It may take time and a lot of effort, can be painful, but it's worth it.
I think this advice by Terrence Tao translates well to other areas: https://terrytao.wordpress.com/career-advice/does-one-have-t...