I understand you want to emphasize business value and short-term returns but barring refactoring is taking it too far. Smart refactoring will pay off immediately by giving you faster turnaround for new features. And fast turnaround is a startup's edge.
I don't refactor for refactoring's sake but I do refactor every time I would benefit right away from the change. It also makes programming more interesting. Once lousy code is refactored, I can go back to solving the right problems and stop fighting with it.