The urban environment represents the greatest challenge.
For starters. "To your door" urban drone delivery would be nearly impossible as most urban dwellers live in condos, apartments or densely populated areas where landing or hovering presents real-world, safety, security and logistical concerns most likely impossible to overcome.
The best case urban scenario is Amazon "drop off points". Basically small Amazon stores strategically located throughout a city with designated landing zones for their drones and close proximity to the recipient - so the recipient (or the delivery person) only needs to walk a few blocks max.
In this scenario you're still looking at a "real person" to get the package to the door though.
"To your door" sub-urban and rural (the outskirts as you call it) in theory may be more doable as dwellings tend to have driveways, yards and space. However the challenges of urban still exist. And landing (hovering) on location, on private property adds greater complexity from a safety and legal perspective.
Amazon has an uphill battle to wage with regards to the legality, rules, regulations and safety of operating unmanned air vehicles over "built up areas" in general. Current pending and existing regulations lean towards flightpaths and fly-zones that ensure drones can land safely in the event of an emergency requiring immediate decent (eg. a crash landing).
This means flightpaths over built-up areas of cities and towns in general would be extremely expensive (funding, insurance, lobbying) and most likely impossible under today's regulations.
No matter how safe they're deemed, drones represent a hazard to persons and property, and as such they'll be highly regulated. Amazon won't be able to fly them anywhere, anytime soon. It will take time.
Now, with all that said, a couple years ago the FAA Modernization and Reform Act was signed, providing funding to the Federal Aviation Authority to establish safety rules that deal with all of the above. The funding as I mentioned has already begun ... to the tune of $60 billion or so.
We'll see how the FAA strikes a balance between corporate interest and public safety.
In the mean time Amazon Drone Delivery is mostly a marketing / PR tactic to generate interest and buzz. Strategic in the sense that, on the surface, it sounds like something the public would like to see. It's exciting! Progress! Get the public on your side and "fast-tracked regulation" becomes much more realistic.