This is something you can get a better sense for when you back Kickstarter projects and see just how much time is spent around packaging and shipping. It is a huge task that a lot of inexperienced companies get totally swamped by.
Edit: I would also like to add that if we bought goods straight from the manufacturer then we would have to deal with greater ship times as well. If you're ordering something from a factory across the US it's going to take a week to get to you. But if Amazon or another marketplace has already purchased the item and stores it a day away from you or in the same city (knowing that there is a chance someone like you is going to order one soon) then you can get the item much faster. That convenience is something that really only a logistics and warehousing middleman can do effectively and people pay for that convenience.
Edit 2: I don't work for or am affiliated with Amazon.com - I just have seen into the belly of the beast at a few online retailers and have good sense of what goes on in there.