The hard part has been coming up with prototypes that go beyond a cool experience, and solve some customer's problem. It doesn't recognize what it's looking at, as much as it's just recognizing the shape of what it's looking at. You have to write the functionality to then determine what it is. Though the HoloToolkit does have scripts that will recognize walls, floors, tables. But it falls really short of the wild ideas my co-workers come up with. There are some example projects out there that will show you how to determine faces, which is cool. But the best ideas most people have, the beta Skype app already knocks it out of the park.
I've yet to test it with two devices, but there's a lot of shared experience functionality in the HoloToolkit.
The Skype call will use the forward facing camera of the Hololens user. For the Hololens user the view of who you are calling will be presented in front of you as you move around. Also, people on the Skype call can interact and draw on objects in front of you.
As an aside, you know you are a real adult when you are given the HL for the first time ever, full of 3-d potential and amazing sound and all the coolest things your 10 year-old-self could think of, and you immediately dive into the wireless and firewall settings.
VR -> Virtual Reality is where the entire landscape is computer generated, the viewer does NOT see anything other than what the computer generates.
AR -> Augmented Reality is where the viewer is able to see around them and the computer projects images to the glass surface, at the right size and color, so that they appear to be in the world around the viewer
Pro-Tip: Want to change medicine and help the world? Find a way to get rid of the wires and tubes. It is really amazing how many wires and tubes there are in a room these days. Yeah, wireless charging and all that, but often that'll mess up a pacemaker (maybe) and so it's avoided. Also, bandwidth issues with a whole building of wireless signals. It's not easy, but it would help more so than a cancer vaccine.