The roads there are twisty, with dry stone walls either side, mostly. Except for the mountain roads that are spectacular.
Then there's the TT, when bikers from all over the world come there to race, and about a dozen or so die each year. Apart from that, though, road fatalities are not exceptional.
Driving there is great, because you don't have the constant worry of "am I going over the limit?" or "what is the limit here?". You stop worrying about the number, or checking your speedo, and concentrate on driving safely on the road. How fast or slow you're going becomes irrelevant, it's all about how you drive on the road you're given.
The speed limit is as much a target as a limit. If the limit is 60 then there's a feeling that "well, they know their business, it must be safe to drive at 60 here". What is authorised becomes advised. No-one can blame you if you lose control due to speed, because you were driving under the speed limit so it must have been OK.
With no limits, it becomes your responsibility to decide what's safe. And yes, there are people who think they can drive safely much faster than they actually can. But they they tend to head for the mountain roads where's less traffic to get in their way. There's also less macho bullshit around it because there's no "rebel cred" in driving over the limit when there is no limit.
Conversely, driving slowly in town is accepted, because it's clearly done for good reasons. Speeding in town isn't socially acceptable (if you want to go fast, head to the mountain and do your worst). People drive more safely, and more courteously because they have responsibility.
It's a great system. I wish we adopted it here in Australia.