Agreed. As you noted, you don’t have much to lose by giving it a try, and you can always drop out and get the MS (like I did).
That's a fairly high cost even if it's fun. It still may be worth doing, and the fun may tip the scales, but... "don't have much to lose" is I think inaccurate.
In Europe, you first study 3 years to get a bachelor's degree, then 2 more years for a master's degree (or an engineer degree), then you apply for a PhD which is supposed to last 3 years.
In natural sciences they do last 3 years in general, in human sciences they are usually quite longer.
I wouldn't bet on that and it's rarely that clean. I think a very common story with PhDs is:
1) after a year or so you have huge doubts you're going to finish
2) you consider quitting but don't want it to look bad on your resume, you don't want to be seen as a quitter, you don't want to let people down, you don't want the money you've spent so far to be for nothing, you know if you give up now you'll never be able to try again etc.
3) you hope it'll get better and push on.
This problem then compounds itself year after year as 2 becomes a increasingly weightier issue and sunk cost fallacy takes over.
Incidentally, 4-6 years is a bit of a wildly optimistic estimate, even if you're not as big a screw-up as I was.