Hacker News Reader

Comment by sircastor | Hacker News Reader

I was just contemplating something like this today as a means of SLAM for a VR headset.

Can the chip read both streams simultaneously, or does it need to multiplex between them?

Hard to say as the firmware isn't in the repo as far as I could see. Most of the ARM camera interfaces are 14 bit parallel buses so it probably has one input. The cameras also output 10 bit parallel so I imagine it multiplexes somehow. You can capture at 60 fps so provided you hardware trigger the sensors you could read out at 30 and still be OK.

If you want something that'll do this, Cypress/Infineon makes a USB3 chip specifically for camera control. I think people have also modified the FX3 for multiple cameras. Omni also have an asic I think, but good luck if you're not selling a billion units.

https://www.infineon.com/cms/en/product/universal-serial-bus...

It's actually very possible to synchronise any number of Pi cameras, to sub-millisecond precision if required. We do this for a machine vision task, measuring the angle of rotating disks at a very precise time (and use an IR flash which is also precisely timed, which avoids any issues with the rolling shutter).

Yeah there are a few "bullet time" demos for clusters of Pis, very cool. Does your application use network based synchronisation, or do you hardware trigger? Whenever I've built computer vision systems it's simple just to hardware trigger everything (no freerunning) and then you just readout the last frame. Almost all image sensors support external trigger inputs, but you need the breakout board to expose that pin (basically all OEM machine vision cameras provide trigger inputs as standard, but they cost more than a Pi+PiCam!).

At IceCube we have 5k sensors synchronised to within a few nanoseconds using GPS fanout and some clever clock distribution algorithms [0]. Cool stuff.

In theory if you use a bright enough flash, the absolute synchronisation of the cameras is less important. For example very high speed photography tends to rely on hardware flash synchronisation with the event. In this case you assume that background illumination is low enough that a many-ms exposure will be black and all your light comes from flash pulse which can easily be sub-ms. Provided the event falls within the camera exposure window, you don't care if there's some uncertainty. You can do (1) camera trigger, start exposure (2) delay to compensate for maximum expected jitter (3) flash triggers (4) brief delay and then stop exposure.

[0] https://www.phys.hawaii.edu/~idlab/taskAndSchedule/ARA/0810....

That's pretty much exactly what we do, we get the cameras into sync using the network then fire a precisely synchronised (us accuracy in our case, so 1000 times worse than yours :) ) a 10 to 150us IR flash to capture the specific details we're after (in our case using retroreflective targets). Main claim to trickery for us is we're worked out a way to sync the Pi cameras in software (i.e. no additional hardware needed).

Very cool to get a reply from someone working in your area, I've followed AMANDA etc in Science and Nature for many years - thanks for the comment!

Look at the schematics. The camera pins on microprocessor is being multiplexed between both camera connectors.

With 1 Mbyte of SRAM and 2 VGA sized pictures inside it will be lots of fun calculating depth map with this microprocessor.

There are Intel Realsense cameras, available off-the-shelf today, which provide stereo vision complete with on-device processing, IR texture projector, and a built-in accelerometer.

If you want to play around with stereo vision and figure out the feasibility of your idea, it could be a good place to start.

They look interesting, but they are being discontinued with a few other product lines as intel refocuses their efforts. Do you know of other alternatives?

There was this discussion a while back when the news about discontinuing came out. It might be a good place to start: https://news.ycombinator.com/item?id=28225065

an STM32 might be a little on the shy side for realtime stereo reconstruction. the OpenCV algorithms i use take quite a bit of time even to process low resolutions images (don't have the number in mind). there's quite a bit of image processing involved to correct for lens distortion and the block matching algos are pretty heavy.

even if camera are multiplex, they might be triggered at the same time (which is essential for stereo).

Reply on news.ycombinator.com