I know what you mean! I've lived in places in Texas that, by all rights[0], should have been walkable, but in practice weren't for the very same reasons:
1) Social inertia: you're going to be the only one doing it, and feel like a pariah. It also feels like you're holding everyone up because they basically expect to be able to turn across a lane that now has a pedestrian.
2) Poor design: you have to take a bizarrely circuitous route that makes you ask, "okay, did anyone actually try using this before final approval?"
Especially sharp contrast going from my office in SF to the Googleplex in Mountain View -- in the former, I could stroll around, take a walk, go wherever. At the Googleplex, you end up having to take unofficial routes between buildings -- walk up a sharp driveway (shared with cars) rather than the out-of-way sidewalk, dart across a busy street while traffic is held up, sometimes waiting on the median for the other side to clear up. Yuck.
[0] Example A: Lived in apartment catty-corner from my (large) university in a small town. Much of the route didn't have a sidewalk, just a curb and some dirt/grass. People would actually ask if you needed a lift! No, just walking to my apartment, 1/3 mile away!
Example B: Lived in an apartment directly across the street (a two-lane, slow one) from a shopping center and less than a half mile from a bar -- but again, drivers don't expect you to cross the street or walk alongside it, and no sidewalk to the bar.
(Just realized this post has a ton of words that are different in British English.)