In my opinion, you mentioned the core reason why you find the excitement baffling, i.e. that you don't drive often but you find it exciting when you do.

Now imagine the scenario for most of the US, a public-transport-hostile country for the most part, where millions upon millions of people burn their precious lives waiting in traffic and sucking in traffic fumes. In my mind, this is one of the most appalling wastes of human potential that has ever existed. Sure, some try to make lemonade out of lemons by educating/informing themselves as they see fit but by and large, it is a huge waste. Not to mention the many thousands of people who die every year in car accidents during the daily commute.

So from my point of view, the self driving car is a thrilling concept: the ability to disengage from a useless, pointless, and hopeless daily grind and engage in something that I want to do, whether it be work, reading, watching a movie, etc. is cool. The closest I have come to this dream in my transit-unfriendly Texas city is the one job where I had an opportunity to take the train/bus into downtown: while this made my daily commute very long, I loved it because it freed me up from the drudge of driving.

Some might ask that perhaps I just hate driving. That is not true. I love taking road trips or autocrossing when I can. But to equate the daily commute with enjoyment is a bridge too far, in my opinion. Banish it, I say, banish it.