No one really goes to a site for the layout and design (unless it's a game or a layout and design site), they go to consume. If there's nothing to consume, but have a lovely responsive layout, they go oh, that's pretty and leave.
Regardless of how ugly something is, if there's an inherent use or appeal in some way, people will flock to it en masse. Case in point: HN.