Surround 360 is now open source | Hacker News Reader

Surround 360 is now open source

code.facebook.com

156 pointssamber31 comments

This is just nerd candy right? I mean facebook is in the same business as google when it comes to what the business model is based on. Get people to pull levers in a virtual casino and sell the resulting statistics to advertisers. The surprising thing is that I think it actually works. New graduates will be lining up to work on projects like this and then end up debugging shitty PHP.

If you get enough nerd candy you can evolve them.

I work on similar projects at Google, so I can say with certainty that this is not just about candy. YouTube makes up a sizable portion of Google's business. Providing 360 capture devices like this (ours is called Jump) allows people to create new content for YouTube, which can then be monetized.

Building cool VR hardware and software is not as distant from Google's and Facebook's core businesses as you believe. Watching 360 videos with 3D audio isn't necessarily the height of human experience, but it's a far cry from "pulling levers in a virtual casino".

And how many people work on that project? 10? 100? 200 people?

Many, and many more on similarly interesting projects. All I can say is that ~nobody writes PHP :)

Lucky then.

> This is just nerd candy right?

I take it that you haven't seen stereoscopic VR video? Although VR is enthusiast-grade right now the excitement surrounding it is by no means exaggerated, whatsoever. I strongly recommend attending a VR roadshow.

Is it virtual casino in your mind first? My first thought is virtual porn.

I was thinking of rats in a maze, running around and pulling levers to get pellets. In the human case I guess the pellet is dopamine or some other neurotransmitter that makes you feel good and is potentially the root cause of compulsive behavior that facebook and google are able to capitalize on.

Hardware looks very similar to https://vr.google.com/jump/ and https://gopro.com/odyssey but with cameras top and bottom.

Really interesting to head about the math and software that went into it though and great that they made it open source!

The GoPro cameras are quite different to the cameras that facebook are using... GoPro cameras have rolling shutter on the lens, whilst the facebook cameras are using higher quality cameras which have no rolling shutter.

The google jump is also using go pro cameras.

Can a rig like this be used to extract a depth map/point cloud, instead of just visual 3d images? What will be the accuracy of such a point cloud?

Yes but not very well. The quality of depth generation output from stereo pair images is pretty tightly linked to sensor size and lens distortion. Camera arrays like this tend to use many low res cameras with wide angle lenses so, again, they are kinda bad for depth generation.

I found a blog full of tools for photogrammetry & depth map creation a while back that are great for getting a better handle on these limitations if you are interested:

http://3dstereophoto.blogspot.com/p/software.html?m=1

You want LIDAR for making the point clouds, and a camera for colorizing them. Pure photogrammetry is doable (I think that's what Earthmine did for a while?), but it isn't as pretty or accurate.

For single-point-of-view, yes. Otherwise, if you can take images from multiple viewpoints, there are a bunch of solutions for pure photogrammetry, eg. Pix4d, Photoscan, Capturing Reality.

Theoretically, there is enough data in the images to build photorealistic models. Algorithmically, we aren't quite there yet.

Considering you can extract a point cloud from regular footage with parallax in it, even from stills, I see why not. It wouldn't be very accurate though.

I suppose you could use infared, similar to how the Xbox Kinect works. However, it'd only be good for indoor use and not as accurate.

I read that Tesla is experimenting with using radar and software enhancements to create an accurate point cloud. (To avoid the necessity of LIDAR)

If you needed it for something real and versatile (industry/production), you're best bet is LIDAR.

I haven't looked at the code for this FB camera, but I'd imagine you'd almost have to rebuild it as stitching images is much different? Seems like you wouldn't have to be as accurate with overlap on the point cloud.

More open source cameras:

http://www.elphel.com/

How about using a dozen Rasperry Pis and camera modules? Perhaps not the same image quality as a $50K system, but is there any technical reason it couldn't work with Surround 360?

How much would it cost to make one?

Just by breezing through the hardware specs, using the stuff they recommend, I'd estimate around $35k -$40k...

So let's call it $50k, built and ready to use =)

those cameras are $1.5k each...17 of them.

Edit: Actually, those lenses are around $700/each as well. And you would definitely want a professional putting it together for you with that kind of money. So $100k probably. Definitely made for industry/production use, not for someone trying to film a weekend trip in the mountains haha.

Retail cost for the Point Grey camera is about 1.3k and Sunex is around $130. Total cost is running about 30-40k, including computer hardware.

>not for someone trying to film a weekend trip in the mountains haha.

The GoPro Omni and the NCTECH IRIS360 may be a little bit closer to the indie scene rather than the professional scene, but those solutions are still $2000 - $5000... and are only 4-6 camera solutions.

There is also 360heros [1][2] which can use up to 14 GoPro cameras for under 10.000.

[1] http://www.360heros.com/2014/01/worlds-first-fully-spherical...

[2] http://shop.360heros.com/3DPRO-360-VIDEO-p/3dpro.htm

Direct source for the cameras requires registration to see the price https://www.ptgrey.com/grasshopper3-41-mp-color-usb3-vision-...

Non-direct source has price listing http://www.edmundoptics.com/cameras/usb-cameras/point-grey-g...

What are the differences between this and Samsung's Project Beyond camera? http://thinktankteam.info/beyond_m/

As an interesting data point, it looks like they designed the hardware using Solidworks 2016:

  https://github.com/facebook/Surround360/blob/master/surround360_design/3d_models/STEP_files/160523_3.STEP#L9-L10

(.STEP files - good for CAD data interchange - often include the name of the creation program)

Nice. Has anyone had a look at the stitching source code - how tightly coupled is it to this particular camera/lens combo? You can pick up Xiaomi Yi's for about $70/a piece.

Of course RAW footage from the point gray will give tons more dynamic range but still would be a relatively decent setup for 3D360.

Pretty sure its not just the raw footage but the face that the action cameras have rolling shutter... I would assume that the rolling shutter along with the cameras not being in sync would mean that doing optical flow wouldn't work correctly in terms of creating a clean image.

Lol guys just buy a gear360, light weight and fits in the palm of your hand. Been using it for months, $350-$400 on Amazon works great!

Gear 360, Kodak SP360 and Ricoh Theta, GoPro Omni are all low resolution monoscopic 360 capture.

Surround 360, Nokia OZO, Project Beyond, GoPro Odyssey are high resolution stereo 360.

Reply on news.ycombinator.com