If I understand it correctly it just uses your GPS and magnetometer to work out where you are, it doesn't actually use the video data like it does for the still pictures you feed it.

Yes.

But as per the other response here, I'm not sure it's using the video so much as the location and orientation to deliver results.

I'd like to see a version, like the still-photos, whereby the search engine is translating in real time as new objects enter the frame.

While I'm at it, I'd like it to perform these tasks with a T-800-style HUD.

Droid indeed.