The Daredevil Camera
ribbonfarm.com
ribbonfarm.com
There's a version which records all the audio channels in real time, using a big disk array. This allows the phased-array processing to be done during playback, so the "aiming" can take place later. The company now has its catalog of surveillance products password-protected, but in the past, there's been more info about that application. The DEA has purchased two.
There are also "acoustic cameras" for locating noise sources.[4]
All this is a spinoff of submarine passive sonar technology from decades ago.
[1] https://www.youtube.com/watch?v=Bepqav87D40 [2] http://www.mathworks.com/help/phased/examples/acoustic-beamf... [3] https://www.youtube.com/watch?v=bgz7Cx-qSFw [4] http://www.acoustic-camera.com/
The system worked in real time but also stored all the raw data so it could be processed again later if you wanted. I also had some fun trying to identify the aircraft based on frequency content and details on engine speeds / blade counts from Type Certificates.
I have an idea for you. You mention that you want to use a webcam so that you can "overlay" this information on top of an image.
What if you could send it directly to your brain?
There is a technology called Brainport[0], which is effectively a 16x16 grid of electrodes small enough to place on your tongue. By feeding electrical impulses through these electrodes, you stimulate the nerves on your tongue. After two minutes of a "champagn bubbling" sensation, your brain rewires the senses from your tongue to your visual cortex and you can "sense" any input.
This technology has been used to give sight to the blind, balance to those with inner ear problems, and sonar to NAVY seals.
It would be amazing if you were the first person in the world to truly "see" sound.
[1]http://www.thinkgeek.com/product/f358/ (sorry, I couldn't find a better source - but the product description explains it)
[1] https://www.indiegogo.com/projects/feelspace-follow-your-gut...
I've actually been collecting the vibration motors from disposable electric toothbrushes to build my own belt; as they were designed to run form a single AA battery powering them should be easy.
I've tried a prototype version of this belt once (for a couple of hours), it worked surprisingly well. But integrating the vibrations as a "new sense" into your brain (so that your subconscious can use it) takes at least some days.
Nicknamed 'batbelt' in-house.
I dropped it after looking into insurance requirements and ADA compliance.
Before I ditched the idea, I was going to put the vibrator belt as a second belt worn under the clothing, but there was no Bluetooth or WiFi or easily accessible way of going wireless between the belts in a practical fashion.
I had my neighbor, blind man who had a guide dog, trial it. He was a great guy, and helped me quite a bit to 'see' past my vision biases. Unfortunately, he passed away. He used to work for the IRS, and we would joke about me getting my deductions past him! He did like what I had accomplished, but many blind people do not have the money for something that is not subsidized, and then you get caught up in the compliance insurance issues. Not sure if that has changed much nowadays.
It was a great maker project before the 'maker' movement. Too bad there was no startup, or crowd-funding web sites back then!
It's not so far from that
Once you've managed to fully adapt to the new mapping, is it just as jarring to revert and go back to normal, or have you learned how to handle both?
Say you rotated for an entire year - or longer - on a weekly schedule between normal vision, vertically mirrored vision, and horizontally mirrored vision. Do you gain full training in all three to the point that you could transition between normal/vertical/horizontal at any time without any disorientation or readjustment period? Or can the brain really only handle one mapping at a time? This would be the ultimate question to answer.
I suspect the results would be similar to how we handle multiple languages. Your primary vision (ie: the one you learned first, or the one you have used exclusively for a decade or more) would remain the easiest to work with, while acquired alternatives may approach the "fluency" of the primary, while not being perfect.
Later on in the video, he mentions that having gotten used to the backwards bike, a regular bike was nearly impossible to ride until he adjusted to it again.
Fascinating stuff.
The thing basically scanned a line across the image, and produced sound based on the brightness. So, with your eyes closed you could quite easily imagine a basic shape of the image from the sound.
Maybe someone would recognize the description.
I understand that the rest of your brain 'trains' this pathway to occur. So because you know that there is a sound source and you can see it move in space, and you 'feel' this on your tongue as well, you are capable of correlating the two senses together and associating them with each other.
And it's about time someone did it! Yours is the first hobbyist project I've encountered since my own extremely crude ten-dollar version in 1981. No square array or processing, mine was a mechanical scanner-disk: eight LEDs glued in a spiral to an old vinyl record album, each with an electret microphone and and op amp. Spin the disk with a motor at about 6Hz. Wind noise, so add foam to the mics. It worked. A mechanical acoustic-phosphor raster sensor and display screen, where sound patterns caused light patterns. The best part was to hold some headphones near it while playing 10KHz sine wave. This produced interference stripes on the scanning disk! I hoped to build a huge version someday, but somebody beat me to it: Tom Zimmerman, an MIT student who patented the dataglove, then used the money to work at Exploratorium Museum for a year (as Artist in Residence), building a scanning-microphone LED disk about 6ft wide.
Since then we have companies selling DIDSON full-blown video for underwater (with sonic illumination,) also a euro company with a 3D phase-array microphone imager positioned on a ring or a wire sphere. See:
http://www.soundmetrics.com/Image-Gallery/Fisheries/Habitat-...
Heh, the divers appear to be skeletons wearing scuba tanks.
http://www.acoustic-camera.com/en/products/microphone-arrays...
It works quite well in my experience with it. The spherical geometry enables a 3D capture and visualization.
Don't miss multi-pages video gallery:
(Unless i missed some sort of karma/like/wow monetization :) )
Wouldn't be fair to people who signed up for something like this camera.
Could there be a version of Patreon, where not the author themself but instead the fans start the pledges. The author could choose to take the money or not, and could prove their identity by putting their bitcoin address in a <meta> tag on their website.
So we could create a project on the site say "support sound camera posts from ribbonfarm.com" and money would be gathered for it, then when a new qualifying post would be made the site owner would be contacted "hey, you have $500 waiting for you, put this meta tag in your <head> to accept it".
This way modest authors wouldn't need to think "am I good enough for Patreon?" and people could support them. People could provide wallet addresses to which the money would be returned if not accepted.
The use case I imagine is finding what part of a machine is emitting a noise, much like how you would use a thermal camera for finding what part of it is running hot.
Imagine a wall of microphones next to a truck - you can tell which part of the wheel makes the sound. That sort of stuff.
As for an actual "camera" that turns sound into an image - i've never heard of one.
SOund Navigation And Ranging: https://en.m.wikipedia.org/wiki/Sonar
Think laser rangefinder on a scanner platform vs a camera.
I'd challenge the notion however that this isn't 'just' a passive SONAR device. Keeping the camera analogy, active SONAR use spread spectrum pings, much like the white light of a camera's flash. Your camera lacks a flash function, and its lens is maybe not yet as in focus as it could be.
If you did want to add a flash, you're likely a $0.02 piezo device and maybe $20 in other various parts (and heaps of numerical processing on a PC) away from having the ability to produce a nice 3D reconstruction of the physical scene.
If you get direct data capture going, please do post some raw data online. I'm sure others would have heaps of fun playing around with it.
One file per cell, 16bit integer samples. Stored row by row, 128 slices per frame.
Not sure where to post it, however - here it would be buried.
This project is similar in that it involves visualization from audio, but apart from that, it's quite difference. This is the ability to see a sound. To derive the positions of arbitrary sounds in a viewing plane, and render it like you would a picture or video.
Most sonar is active (sends out pings) and has only one input (one microphone)
It builds pictures by directing the ping and measuring time until the response.
This is fundamentally different than an array of microphones passively listening to, and creating a fine grained picture of some sound source.
While I believe the US Navy doesn't prefer the term "Passive Sonar", submarines and some surface ships are equipped with hydrophone arrays (either on the hull, or towed arrays, or both) as well as active sonar; this isn't an uncommon thing. (I was under the impression that the active sonar didn't use a single hydrophone, but instead used the same hydrophone array used for passive detection.)
http://blueprintsubsea.com/starfish/images/sonar/452/sf452_s...
That's just one example - look for side scan sonars for the earliest stuff (https://en.wikipedia.org/wiki/Side-scan_sonar) or sonar imaging for the latest stuff (http://www.tritech.co.uk/product/gemini-720i-300m-multibeam-...)
For my undergraduate project years ago I built a microphone array that could be used to locate noisy objects. I demonstrated it by tracking aircraft taking off at a local airport.
Whilst I didn't make 'acoustic photos' with it, I did plot the flight track of the aircraft in real time, and could have a decent crack at identifying it based on frequency content.
I have. His name is Daniel Kish [1, 2].
Apparently his visual cortex has learned to process sound rather then light, so he literally sees, based on the sounds he is hearing.
Your point about only him appreciating his own experience is interesting, as the same question can be asked of all of us [1]. In audio, maybe apples and oranges are the same colour and strawberries are different? It would be an interesting application of a sonic camera, to render the image so that equivalent audio colours are consistently mapped to the same light colours.
[1] https://www.quora.com/Visual-Perception-Is-everyones-experie...
Cool stuff!
I'm unaffiliated, but saw one of their demos.
- Kai @ nlacoustics.com
The use case here is being able for a diver or ROV to see in murky water
[0]http://www.soundmetrics.com/image-gallery [1]http://www.apl.washington.edu/programs/z_DIDSON/Pages/LIMIS....
It's used my military & events for drone tracking by sound, and concert venues for seeing where sound leaks out to the surrounding neighborhood. (Among other uses)
Edit: Their example screenshot is even showing an engine block overlaid with acoustic output of the components: http://static1.1.sqspcdn.com/static/f/1431009/18268254/13373...
(downsampled back to 16x16, than upsampled with photoshops more sophisticated resize and doubled speed for more fluent movement)
The address is on my Orbital Designs site - http://orbides.org/
A fund that hackers can apply to for building cool projects is an interesting idea though (to the extend that I'm sure one already exists).
Kudos to that!
you mean a $10 EZ-USB FX2LP dongle?
You might be interested in this https://sourceforge.net/p/manyears/wiki/Main_Page/
Im sure you know about SeeSV-S205, its basically the same thing, but polished and packaged neatly for FAT commercial customers ($30K).
posted on slashdot in 2013 about the same thing https://tech.slashdot.org/comments.pl?sid=3742015&cid=437167...
Modern i7 cpu has surprising amount of flops and could handle this no problem (not to mention modern gpus). Consider modifying one of your boards, instead, or in addition to doing fft on fpga try pumping raw data over usb to a PC, 64 3MHz PDM streams should be 25MB/s? comfortably within usb 2.0 limits. FX2LP breaks no sweat while pumping ~40MB/s
top ebay link $6 http://www.ebay.com/itm/CY7C68013A-56-EZ-USB-FX2LP-USB2-0-De... application: http://www.cypress.com/documentation/application-notes/an613...
That's "64 3MHz PDM" per cell, of which there should be 16. And, they have to be synced to a sub-microsecond precision.
If all you want is the current setup then above is valid with 2x $6 modules.
This is also why you can't just plug four stereo cards into a PC and expect it to make an 8-channel system.
Huh, I always sorta wondered about this. Thanks for clearing it up!
You could make a long multitrack recording this way, and so long as you played it back from the start on the same hardware (i.e. so the soundcards would drift relative to one another more-or-less consistently with how they drifted during the recording), the playback might be ok. But if you begin playback half way through the recording, then the timing between channels might be out, as your software has no idea how much drift each channel ought to have accumulated by that point in the recording.
Mind, it's been at least 15 years since I actually put any of this stuff to the test. Tolerances on modern soundcards might be really tight.
Shouldn't 3 mics be somewhat sufficient for intermediate values as well? More mics => more redundancy => clearer image of the waves, but why do the pixels/blobs "snap" to a microphone?
edit: ah, he does not use timing (at 10fps obviously not) but needs continuous sound to see the wave pattern across the array...
edit 2: so do these gunfire locators¹ only time stamp impulses? Or do they do continuous analysis? With natural, irregular sound, it should be possible to match and measure the delay between mics, no?
edit 3: the gunfire locators would need to analyze for gunfire characteristics anyways, to weed out false positives, but that could be separated from the locating.
edit x: oh cool, if you zoom out of the final picture so it's smaller on screen, you can even see some parallax effect happening: http://206hwf3fj4w52u3br03fi242.wpengine.netdna-cdn.com/wp-c...
¹ http://wordpress.mrreid.org/wp-content/uploads/2013/06/boome...
It turns out that noisy machines are often mostly noisy because a relatively small number of parts are too flexible, not screwed on tightly enough, and so on. Track that down and you know where your low-hanging fruit is. A daredevil cam is exactly what you need for tracking that down.
Their site is a tad unimpressive sadly, but their tech is very cool: http://www.sorama.eu
Yep, that looks very close to what i'm making.
While such a contraption should, in theory, be able to distinguish between dozens of voices in a room, it would take much, much heavier signal processing to achieve.
Consider that most of us are carrying around multiple, networked mics. Some may already do it, but I doubt it will be too long before everyone can virtually tour real-life spaces and visualize the sounds there in real-time.
Interestingly it's also doable with a single mic - previously with algorithms like NMF and ICA, more recently with neural networks. Our brains do it very well so there is probably more to discover.
Google blind source separation or cocktail party problem if you want to find out more.
http://www.npr.org/programs/invisibilia/378577902/how-to-bec...
Nitpick; Duga 3 was never built. The one near Chernobyl is Duga 1.
But I didn't mean to detract, that's really a very cool project that you did!
If you get a x86 hooked up, doing some software cleanup might make the video look much better. Some ideas that you'll probably come up with on your anyway:
- Assign colors based on frequency clustering, rather than a heatmap (so, red=first freq, green=second freq, and so on).
- Persist colors if the frequency in a region shifts slowly enough (so every source stays mostly the same color).
- Remove echoes by finding correlated pixels.
https://www.newscientist.com/article/mg21929364-400-matchsti...
Regardless of commercial application, this guy can do #hardtech
Now is the reverse (vision->sound) also possible? Perhaps using an array of laser sensors, so that we can reproduce the sounds in the environment.
Also, is it possible to turn the images/video back into sound? Could be a fun experiment!
Listen to this with headphones: https://www.youtube.com/watch?v=IUDTlvagjJA
It’s amazing how much you can geolocate just in even a recording.
https://www.flickr.com/photos/fotoopa_hs/sets/72157611107153...
https://www.flickr.com/photos/fotoopa_hs/4082464167/in/album...
You really need a spotter to find out which of your projects are cool and which aren't.
Then again, after trying to post a dozen times and getting nothing, one would just stop trying most of the time, and would work for the attic, so to speak.