Midtown Manhattan is a sinfully ugly tourist trap, especially around Penn. I'd argue it's the worst neighborhood by far. It's a twinkle in tourist's eyes but I avoid it like the plague.

The thing about Penn too is that it's extremely difficult to calculate your bearings. There's no significant landmarks to orient yourself by - every corridor looks the same. I use Penn regularly and I occasionally get lost if I happen to use a different entrance.