Is added depth information really a game-changer? People have already used 2D, photosphere-style images to do what you're suggesting for streets and places visible on StreetView. [1] I understand that more information is going to be helpful, but are there many scenarios that significantly benefit from a depth map to judge accessibility?
Combining crowdsourcing and google street view to identify street-level accessibility problems: http://dl.acm.org/citation.cfm?id=2470744