For most startups, they are hoping to get some money (investment, revenue, etc) in order to keep themselves going in the first bits. Buying servers is a huge capital expense, and not all startups have excellent credit and ability to do long term financing. But almost every startup will have access to a credit card and be able to pay a (reasonable) cloud bill on a monthly basis.
I've worked with a business that wanted to put TBs of data (with monthly growth) into AWS. I think the break-even point to buying a fully automated 10 tape library was something like 14 months, by which time they would have fully paid for ALL the tapes, the changer, and an extra drive to do emergency recovery offsite.
I think with startups it's about controlling costs -- developers are expensive but needed, sys-admin skills are expensive but "easily avoided" by just using AWS. Some of it is also the ease factor, where you can use a bunch of cloud services to do things that are powerful, at fractions of a penny per transaction.
I also think some biz people aren't always tech experts, and tech people tend to either want to try something shiny (to pad their resume) or stick with something they know well. Cloud and AWS particularly make that easy.