Show HN: NeuralCam Live – Using ML to Turn iPhones into Smart Webcams | Hacker News Reader

Show HN: NeuralCam Live – Using ML to Turn iPhones into Smart Webcams

neural.cam

117 pointssoonpls64 comments

Nice work!

I have been researching and working on this webcam quality problem for a while. I have tried a couple of webcam-over-phone solutions and while they solved the image quality issue, they tend to introduced quite some lag. Especially when you use your dedicated microphone, then the lag becomes a huge issue (because the microphone has minimal lag and the webcam has a significant lag leading to the voice not matching the lips).

Have you thought of the issue and have you done an analysis on that?

Besides, with this setup, you have to rely on a driver that turns the video from whatever phone to a webcam. That in my experience is quite flaky, but perhaps not inherently problematic. So I wonder if you could comment on OS support for people who want to use your software with say, Linux?

Disclaimer: I use and package a software suite to turn a raspberry pi high quality camera to a webcam. The lag seems pretty good and it appears as just a good old usb webcam on linux and windows (although I haven’t managed to make it work on macOS) [1].

1: https://github.com/showmewebcam/showmewebcam

thanks, jimmies! we do not have a dedicated microphone feature at the moment, and we did encounter a small amount of lag, but it varies based on device model. we hope it won't be an issue for our users. regarding the setup, we're aware of the driver dependencies, we're planning to build a complete Mac app instead of the plugin we have now, and to add Windows support soon as well. about Linux, it's not supported at this point but on the long-term, we want to have compatibility with all operating systems.

I wonder how hard it would be to get a MIPI capture device that presents v4l2, I guess it's almost certainly more expensive right now than a Pi Zero.

Anyone know of a SDI or MIPI capture card that works off the shelf with v4l2?

I haven’t tried looking for it and it seems like minimizing latency on the pi 0 is more fruitful and flexible than trusting such an adapter to work. And as you said, cheaper.

Note for anybody thinking they want to use this with Zoom/Slack/Discord/etc: It appears that this has the same restrictions that other DAL-based apps have and you'll need to re-sign the app or disable SIP to use it that way. I've been using an OBS virtual camera for this purpose and have had to do this a few times now. It will depend on the target app if they have this in their allowed plugins list.

https://github.com/johnboiles/obs-mac-virtualcam/wiki/Compat...

On the SIP workaround, it says disabling SIP for the filesystem is enough, but even though that's less broad then disabling SIP entirely it's still a pretty wide brush. Have you (or anyone else) tested whether it's enough to disable SIP just for specific applications or folder locations? IIRC, all definitions and exceptions for SIP fs restrictions are still in two locations:

  /System/Library/Sandbox/rootless.conf

  /System/Library/Sandbox/Compatibility.bundle/Contents/Resources/paths

Disabling SIP entirely allows editing these as root, and then reenabling allows more fine grained control. Of course like everything else in an Apple-owned location, changes have a high chance of being blown away any full system update, but those are rare enough (and always require a reboot anyway) that it doesn't seem like a major pain to just redo changes a few times a year (with a script).

There are still of course some security implications for any additional exceptions you add but a lot less then a full fs disable. Further it's cool to have control over it anyway because you can even extend SIP to additional locations of your own which could be handy for some scenarios.

I can't comment to disabling SIP as I haven't taken that route. I've generally taken the codesign route instead, which does require re-signing the app after every update. In the case of Zoom, that's gone away in recent versions since the app now has the virtualcams I'm using in its whitelist. For anything else, it's a very manual process.

Quality enhancement of the face region is to my knowledge the best use regarding ML and real time communication.

The idea is to detect the region where the user's face is and then transmit that region at a higher quality than the rest of the video image.

This vastly improves the video call experience as the face is almost always the most important part of any video call.

Yes, face is the most important part of most video.

80% of the intellectual content of most telepresence is transmitted via audio. That's why when your local TV news goes to a live cross and there's no sound, they go back to the studio until the audio is fixed.

thanks for your input! at the moment, we have a couple of features that improve the face region quality, but we'll be adding more complex features in the future as well.

I've tried EpocCam Pro [1] which is good at the basics. The biggest challenge with these "phone as webcam" setups is getting a good mounting system. A tripod is hard to get in the right spot, and stacks of books are fail. Neither EpocCam or this software solution mention mounting which makes me think they're all not putting themselves in their user's shoes. My sense is the team that wins here get the "basics" right of this -- flawless connection to computer, good awareness of Do Not Disturb/it's a phone, and suggestions for mounting. Very practical. Phone camera is way better than webcam so the rest takes care of itself.

[1] https://www.kinoni.com/

3d print an edison bulb that can clamp on to the phone, then you can use an adjustable Ikea lamp at the camera mount. Might jiggle a bit though depending on your setup.

They're not cheap, but I hear the Mountie[0] works well for physically mounting your phone (or iPad) to your laptop screen.

[0] https://tenonedesign.com/mountie.php

Update: I have since decided to purchase a Mountie and I haven't used it much yet, but I'm pretty happy.

there are a ton of phone mounts on amazon, it depends on where you need the phone. the desk clamp+goose neck ones are quite versatile, sometimes a simple phone stand like this is ok https://www.amazon.co.uk/dp/B07DHH95V1/

I tried EpocCam but it wasn’t great and the free version has limited resolution.

Now using Iriun webcam which looks a bit dodgy but actually works well over wifi. Completely free.

There’s also an app called Camo which probably is the best technically, but I dislike the recurring yearly price. One-off price seems reasonable but why do I have to keep paying assuming I’m happy with the app and require no upgrades? There’s no hosting...

I got a ~17eur stand on amazon called Lamocall or something and it’s really great! (Sorry for lack of links but I’m on mobile atm). Overall I’m happy to make use of my wife’s old iPhone 6 and get a better web cam quality.

The main nuisance though is that you cannot stop and start the app from the computer like a normal webcam. So it’s always on or I have to reach out. Since I’m using the better-quality back camera of the phone, the screen is facing away so it’s very awkward...

I've recently found a mount for iPhone XS Max - it's the paper ring from duck tape... Works very well for facetiming between bubs and paps.

This looks awesome - unfortunately, my 2017 MBP running the latest MacOS 10.15 and iPhone 11 don't seem to register with this software when plugged in after installing the app and drivers. The Support Center doesn't have anything useful for troubleshooting, so...YMMV?

Same. MacBook Pro 16 & iPhone 11 Pro.

Video source didn’t show up in Zoom but did in QuickTime, however there was no video - only a static turquoise image.

Had to faff about parsing the Mac installer with pkgutil once I decided to remove it as there were no uninstall instructions or mention of what was installed where.

Not sure if it's any help but with Installer open go to File > Show Files and that should show you what's being installed to help you manually remove them.

Make sure the app is running on your iPhone and your iPhone is connected to your Mac with a USB cable before you select the virtual webcam in video call apps. Regarding Zoom support, the latest version Zoom version (5.2.0) should work with no issues. If you're still encountering any sort of issues, send us an email at support@neural.cam and we'll look into it asap.

We'll be updating the Support Center with full troubleshooting information until tomorrow. In the meantime, try opening up a video call in Chrome / Firefox (as Safari does not allow any kind of virtual webcams) and if your iPhone is connected with an USB cable to your Mac, and you installed the plugin, you shouldn't have any issues using the app as a virtual webcam.

Thanks for the quick support. Should've mentioned I'm using Zoom. Updating Zoom to v5.2.0 did the trick for me.

awesome, hope it works well and it brings value to your video call experience :)

Well, sounds interesting. I've heard some people use old smartphones as cheap webcams & LTE modems.

Yes, I have been using DroidCam and its been great. I am using my 3 year old phone for this and the camera quality is still better than the laptop webcam bought last year

Probably can use as some sort of server too. Quite amazing considering power use and cost.

What's the pitch for the subscription model for NeuralCam Live +PLUS other than "New Styles added regularly" (which I don't think I care about, although I've used the product for less than a day so maybe with time)? The other +PLUS features (additional video styles, special soften style, low light mode, no ads promoting other NeuralCam Apps) seem like one-time purchases, not something that as a consumer I could justify paying a recurring for.

we want to make this app usable for most people in the free version too. we see styles as the core component of the overall UX and being able to select the right style for a given setting is quite useful. we think that many users will appreciate a wider choice. we’ll be adding additional features to the Plus version in the future while also continuously improving the tech.

Got it, thanks for responding. So my takeaway is that today the pitch for the recurring is that you'll continue to add new styles, and users can pay a monthly to access those as they come out. Just one potential customer's opinion, but I'd be much more likely to purchase styles and +PLUS features if they were one-time purchases.

Not speaking for NeuralCam, but for big tech the business model for this kind of software would be surveillance capitalism, which is why they will offer it "free".

https://www.youtube.com/watch?v=hIXhnWUmMvw

For that reason, I'd expect NeuralCam to end up as an acquisition, depending on their motivation, resources, and all the rest.

Looks good OP, just a suggestion, I opened the app and continued through (clicked continue to live feed) without airdropping the driver or clicking the link. Now i can't find a way to get back to that screen on the app. There should be a way to go back to that onboarding so that the user can both preview the image quality without having completed the setup but also continue it later. Just a suggestion! I'll try it later.

thanks for the feedback, juliann. we'll be adding a "restart setup" button in Settings. in the meantime, you can download the plugin directly from here: https://neural.cam/live/NeuralCamLive-installer-1.0.0.pkg

BTW your installer includes a gitkeep file.

Genuine question for people: When I read about things like gesture guard, head bubble, virtual backgrounds, etc., I cant help but wonder why they’re so important.

Is it really a problem if you sneeze or cough during an online meeting? Are you colleagues really making this an issue?

Likewise, I’m fascinated by the focus (ha ha) on hiding backgrounds. Why is it so bad to show people where you are?

I’m clearly in the minority in that neither of those scenarios are even remotely (ha ha) bothersome to me or my colleagues. But from their overall thematic popularity, obviously these are real issues for people.

I’d love to be enlightened...

Many people don't have a room in their house or apartment that they can dedicate to creating a home office, especially people with kids, and have to use a shared room. Blocking the background is essential. Also, sneezing during a pandemic sounds like something that could get an employee or contractor a furlough.

Haven't gotten to try it yet, but is head bubble actually black around the edges, or is that just to show off how it works?

First thought was it would be nice to combine head bubble with blur, so you avoid the stark background -- that's probably already what you're doing though. Will give it a shot later today.

yes, the Head Bubble puts your head in a bubble, making your background black and tracking your head. it's a great way to hide your background and it really takes away any distractions from video calls.

guessbest replied perfectly regarding the background question.

about the gesture guard, we believe there are many scenarios in which it would be useful. We realize sneezing or coughing might not be the worst to witness during a video call, but other actions like picking your nose, biting your nails, etc might be perceived as inappropriate, so we're aiming not just to hide those moments, but to help users get rid of the bad habits on the long-run. and then there's the nudity detector, which I covered in one of the other comments.

I tried the software but was so put off by the ads in the app and the inability to really try the app out before paying. I they had a 10 day full trial I would be in a better position to decide if I buy or not.

hey xutopia, thanks for the feedback. the only ad in the app is towards our other app, NeuralCam NightMode. we do have a 7-day trial option available, you can try out all the features with it and cancel at any time if they're don't bring you enough value.

Wow. Looks interesting. I'm curious, why don't you use the built-in Macbook webcam? - does Apple limit its use?

The Macbooks webcam is a lot lower in quality when compared to even older iPhones, so the image quality would be really low. Also, Macbooks do not have a Neural Engine at the moment, so running advanced machine learning algorithms in real-time is not really an option, when compared with the low quality webcam. The iPhones & iPads, though, have both a Neural Engine and a very high quality camera, and we're making full use of them with our solution.

You mention background segmentation - is there any way to replicate a shallow depth of field/large aperture with the iPhone's dual lenses?

I'm sure this is quite challenging to get the segmentation accurate in real-time.

We'll be looking to build these types of features in the near future, as our engine evolves. It's challenging yes, but not impossible.

An interesting idea (which would need to be integrated in the conferencing apps) might be too do quality improvement on the receiver side. Turning low bandwidth and compressed video into higher quality video.

This sounds interesting - but how are people setting this up? I don't think i'd be able to use this out of the box, since i don't have a stand for my phone to position it like a webcam.

> Gesture Guard

> Detects and blurs different types of gestures that don't have their place in a video call.

> Currently, it has two main components: detecting and blurring face touches (scratching face, picking nose, yawning, etc.) and detecting nudity. These help you to not get caught off guard.

Finally I can do my stand up in the shower.

This sort of feels like an April fools page.

The 'nudity' scenario is somewhere along the lines of you being in a video call, and your loved one walking naked in the background, or just forgetting to close the webcam while you change, etc.

We realize it's not a common scenario, but even if it would work once or twice per year, it would still be saving you from a very awkward situation, thus the value we see in this.

Politician offers to resign after showering during live video meeting

https://www.independent.co.uk/news/world/europe/councillor-s...

Pretty wholesome.

Unfortunately he believed minimizing the app would cut the video.

There's no substitute for a hardwired camera-on indicator light.

Otherwise politicians are going to keep showering on camera during meetings, instead of showering off-camera during meetings, like they're supposed to.

I know that the webcam USB API requires the on-off light to be controlled in software, but can't webcams just... Ignore that, and just control it in hardware? It seems to me that would be better for everyone. Heck, we could mandate webcams have hardware indicator lights by law. You can always tape over them if they bother you.

It's kinda a mix of things. Webcams built with IR sensors and emitters (like the Logitech BRIO 4K) don't use the indicator while using the IR, though you can clearly see the pulsating near-infrared LED while it's on.

On Linux at least, if you open the v4l2 device for a device like this, it will turn on the indicator directly, and I don't think you can attach to it with libusb while it's otherwise in use by v4l2. In practice, I think the worst you could do from userspace on Linux is show the indicator while not recording.

I honestly don't see why he would offer to resign, seems totally innocent?

Another scenario: https://www.bbc.co.uk/news/uk-52835351

Blurring it out so that people can tell what you’re doing, but not see the details? I don’t think that’s much better, honestly.

"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."

https://news.ycombinator.com/newsguidelines.html

Please read the Show HN rules as well: https://news.ycombinator.com/showhn.html. They contain additional guidelines for how to react to someone sharing their work on HN.

It was just a joke. The April fools part was in relation to the novel ideas that are presented that this app allows for. If anything it is a compliment.

NeuroCam Returns! https://www.youtube.com/watch?v=q9XmXad_wmo

at first this looked like a spammy product advertisement page, but actually, there's some interesting tech here! I'd suggest you make things like face bubble, low light, gesture guard and denoising more prominent for the HN audience, instead of "marketing material" like your future products that don't yet exist.

thanks for the feedback, tomp. we didn't tailor the article for a specific audience, we tried to be as transparent as possible with our vision & mission. we'll think of ways of improving communication both for casual users and users like the HN audience.

Dare I ask about Android support?

yes you may :) however, Android is much more fragmented platform and it’s only really possible to run the advanced ML algorithms we’re using on the newest Android phones. so.. we’ll probably have some level of Android support in the future, but it’s not clear which of the features on which devices.

Reply on news.ycombinator.com