Hacker News Reader

Comment by cubefox | Hacker News Reader

It's strange that in the age of AI, denoisers are still so bad. It's basically impossible to photograph snowing in the winter because the denoiser will remove 90% of the snowflakes. Machine learning models are already used for denoising ray traced graphics with substantially improved results, so why is it that cameras aren't using ML denoisers yet? At least for still images. Or do they perhaps already use them, only the quality is still bad for unknown reasons?

View on HN

(As someone who worked closely with pathtracing renderers and de-noisers, I think I can answer this :) )

It's mostly because in the VFX/CG space for ray tracing/path tracing de-noisers, they almost always rely on extra outputs/AOVs of things like 'albedo' (diffuse reflectance), normal / world position, etc, to help guide them in many cases.

So they often can 'cheat' a bit, and know where the edges of things are (because say the object ID AOV changes - minus pixel filtering, which complicates things a bit).

They can also 'cheat' in other ways, by mixing back in some of the diffuse texture detail that the denoiser might have removed from the 'albedo' AOV channel.

Cameras don't really have anything to guide them, so they have to guess. And often, they seem to use very primitive methods like bi-lateral filters (or at least things which look very similar), to try and guide them, but it doesn't work very well.

Portrait cameras on phones can use depth sensors a bit to help if the camera has them, but for things like hair strands, it doesn't really work, and is mostly useful for fake-depth-of-field depth-based blurring.

Yeah, but surely ML models would at least work better than analytic algorithms. After all, when looking at a noisy picture, our brain is pretty good at distinguishing detail from noise, so it's not clear to me why an ML model couldn't have denoising performance similar to the human brain, even if it doesn't match the "cheating" denoisers used in ray tracing.

It would probably help you to compare what you can do on a phone vs what you can do with desktop software (Lightroom/Photoshop, DxO, Topaz, CaptureOne, etc). It's generally quite good, with the exception of challenging liminal areas (e.g. hair, foliage).

Fwiw, Topaz -- which I have a license for but essentially never use -- has pretty incredible denoising & upsizing features (for both photo & video), but to get the optimal quality output you offload the processing to their cloud infra (and buy credits from them to pay for it). It's roughly the equivalent of a SWE using a local LLM that's good enough" vs a frontier model that's SOTA but requires a consumption-based subscription.

Interesting, so it seems to be an issue with heavy compute or RAM requirements.

I've got a 61mp camera, and an RX 7900XT. It takes about 15s/picture for DXO to denoise, which is a lot longer than people are willing to wait on a phone to take a photo. Topaz is even slower. A cloud service could be used to do it in post, but someone has to pay for that.

Yet in modern computer games, modern graphics cards denoise a scene in real-time at 60 frames per second using machine learning models [1][2] while doing all the other rendering at the same time. Granted, that's ray tracing, and the resolution is lower, and they technically cheat by using additional information, but it might be that DXO is not optimized very well.

1: https://blogs.nvidia.com/blog/ai-decoded-ray-reconstruction/

2: https://gpuopen.com/amd-fsr-rayregeneration/

Games will typically un at 4k or less which is about 8mp on the other hand it’s difficult to buy a stills camera with less than 20 mp, and >40mp is common. Most algorithms are n^2 in graphics as well so we wouldn’t expect a linear speed up. I’ve tried dxo, Lightroom, and topaz they all perform about the same so I don’t think it’s particularly unoptimized.

Would you be happy if ML detects that you are trying to capture the moon and replaced your shot with a memorized picture of the moon with match ing size and orientation ?

Autoencoders can presumably do this except that it would be operating at levels of patches rather than the entire Moon.

No would be unhappy :(

ML models on some Samsung phones did exactly that.

https://news.ycombinator.com/item?id=35107601

It was probably a result of an explicit company charter but such things can happen with modern ML models quite unintentionally.

> Yeah, but surely ML models would at least work better than analytic algorithms.

Bear in mind, a modern flagship phone doesn't just need to take photos - it also needs to record 4k video at 60fps.

Can't go too hard with the ML when you've only got 1/60th of a second to do it.

I don't think this is even just an AI denoiser problem. Rain effectively looks like haze for the parts of the scene that are at sufficiently long distances. Therefore it's difficult to use dehaze with rainy scenes. The harder the rain is falling, the more limited your ability to dehaze. It's much the same with denoise. You won't be denoising at 100% in Topaz if it's dark and raining hard.

One day smartphone cameras might get there, but right now, the sensor technology isn't there yet. The problem isn't merely noise. It's that rain and snow are moving about in the scene, which means that the camera can't just do its usual trick of taking multiple exposures.

AI denoisers are nice, but they aren't strictly necessary for ILCs. Even those full-frame cameras without IBIS are able to take pictures of nighttime snow just fine.

Are we still talking about smartphone cameras? If yes, apps already heavily rely on much more advanced computational photography than your average photo editor can do, including but not limited to ML denoisers. The problem is that such apps are typically optimized for the "average case" and are as automated as possible, so they either remove snow, rain, and haze intentionally, or lose small moving particles as the result of stacking. That said, snow and rain are usually possible to capture in the apps that attempt to determine the scene type or have specific modes.

Would you have an example of an app which can photograph snow and rain?

Reply on news.ycombinator.com