I believe Paul Graham once defined the difference as: a startup wants to (but doesn't necessarily) have "a significant exit," it either goes public, or gets acquired for a large amount. A new small business that aims to be sustainable, profitable, grow, etc, isn't a "startup," without the significant exit. Unfortunately I remember hearing this in a video or audio interview, so it's hard for me to search for this attribution.