...assuming the startup is bootstrapping. Why?
1) Ugly designs are often very effective. 2) You can polish it later when you're making money.
Your site might look amazing. Your site might look crappy. If no one knows about your site or no one cares, then it doesn't matter either way.
Focus on traction for the product or service you're providing, then worry about the design of your letterhead, business cards, shiny web 2.0 logo, etc.
I'm hoping it's true because I'm bootstrapping an app that's on the ugly side. I've been told the product and workflow are solid but that the amateur design will hurt conversions. It's a B2B product. Personally I think the ugly design will filter out some tire-kickers (which is good because the product is somewhat compute-intensive) but anyone who knows they need the product will not be deterred. Of course I could be biased.
I'd like to add that I don't intend to keep it ugly, but I'm hoping to see some revenue first.
An ugly design that works is actually a good design.
Several people in this thread make the point that design isn't just the veneer, but definitely the whole user experience. But sometimes people confuse "looking pretty" with "well designed". Those often go hand-in-hand, but not always, which is very important to keep in mind.
It's easy for folks who are used to conflating the two to accept a good looking design because it looks good, rather than it working well for the user.
That said, for a consumer facing product, user experience and design does become important after a point - but it need not be your priority at the very early stage.