Hacker News Reader

That is completely wrong.

For foveated rendering one could easily build dedicate hardware into the device which would require almost no additional power. And those algorithms are not that expensive to begin with. Every camera has face tracking build in (which basically is the same problem) and that does not drain the battery either.

And why should northern Europeans eyes work differently?

Even if that would be true one could add a bunch of settings for personalisation.

Eye distance for example is a common setting on dodays devices...

Eye trackers typically require 2-4 additional cameras trained on the eyes. The algorithms for eye tracking are typically pretty involved, but maybe you know some very lower power algorithm to track the eye at 120 hz? I'm not sure how dedicated hardware will reduce the power consumption for 2 cameras and an ML pipeline to almost nothing.

Northern Europeans have very light irises. It makes it easier to track the pupil from a single signal-to-noise perspective. But there are a ton of weird ethnic challenges for eye tracking. Things like reflectivity and moisture of eyes varies a lot across different across different ethnic groups. Personalization add complexity and will only take you so far when you are dealing with physical sensing challenges.

Eye-tracking uses simple IR reflection sensors to follow the black dot of the pupil. It has no dependence on skin or iris color. It's been done for decades in psychological studies and doesn't require complex ML algorithms to do. Eye tracking is not a particularly hard thing to implement in hardware, it's just an added expense for dubious--very dubious--utility.

Probably the "best" use of eye tracking for modern VR systems is pretty much just reflecting the eye motion in avatars during VR teleconferencing sessions to give a more natural "face-to-face conversation" feel. Beyond that, things like eye-tracking as input and foveated rendering turn out to only be great on paper.

>Eye-tracking uses simple IR reflection sensors

Every VR headset with eye tracking uses actual cameras.

These days, wouldn't a camera sensor be considered a simple sensor? Optical mice use actual cameras too, they're just optimized for a very shallow focus, very high temporal resolution rather than spatial resolution, and sensitivity to IR. Do eye trackers need especially high spatial resolution, or RGB color?

I was mainly referring to the power requirements.

You are underestimating cost of rendering and overestimating cost of machine learning models. And forgetting the fact that cameras are already used in modern headsets for position tracking and more(there is Quest 2 demo where you cast spells using your hands, no controllers), and it is much much more complicated task than eye tracking.

If you have some numbers please share.

I've seen NVIDIA and Tobii (highly motivated parties) claim that you might get a 10% power reduction, but those studies were based on very constrained operating conditions and conveniently brush certain details under the rug. All told, it's a lot of work and a lot complexity to chase something that will at most provide modest gains.

The cameras on headsets face the world. So you need 2-4 additional cameras trained on the eyes. These camera each require power just to get images. For a decent UX you need tracking at ~120hz (60hz eye tracking is slow). Exact figures are hard to come by on a Sunday morning, but off the top of my head, I've seen figures to the tune of 250mW per camera (which ignores ISP and the compute to move the data around). Computer vision algorithms running at 120 hz are typically not cheap. Pupil Labs cites 50% CPU (on an i5) per eye[1]. Tobii doesn't publish their power consumption but it is also pretty steep. I'm sure there are optimizations here but all the specs I've seen blush at the power consumption required by a general purpose eye tracker.

If you have figures that suggest otherwise I would be happy to see but every quantitative analysis I've seen comes up with a muddled response. If you already commit to running eye tracking for other reasons then it makes sense to run this rendering optimization, but I'm not sure if it really pays for its self. [1] https://pupil-labs.com/products/vr-ar/tech-specs/

>And forgetting the fact that cameras are already used in modern headsets for position tracking and more

Just because you already are running cameras + ML models that doesn't mean adding even more cameras and ML models is free.

I would think it's actually harder to do eye tracked foveated rendering than motion tracking. They're not identical problems. Motion tracking is a feature, foveated rendering is an optimization that needs to significantly beat the naive approach of fixed foveated rendering.

Part of the problem is you can throw everything you have at motion tracking. You can over render and then use the end of frame head position to space/time warp to adjust the render the display gets, maybe throwing away some of the edges.

For foveated rendering, you need to know what to render at the START of the frame before you render it, there can't be any waste, and we're only talking about a thin band of rendering gain IF the user is not looking forward. I would assume the focal point can change faster because you're at least dealing with the head movement + eye movement, so it's even more work that needs to catch up at the end of the frame.

I really do not think foveated rendering is easier than head tracking. If it was, we would see more of it. Dynamic foveation is really only just now coming to headsets.

While we lack spatial resolution in the periphery, we can be very sensitive to movement or changes there. So I think you would need a pretty accurate and stable low resolution rendering method to avoid inducing unintended perceptions of movement.

I assume saving power means skipping a lot of processing, which sounds to me like undersampling i.e. the opposite of the supersampling used to get good renderings today. But, it is hard for me to imagine how a generic 3D modeled scene can be undersampled for the periphery without introducing horribly unnatural aliasing artifacts. In other words, you still need to determine proportional occlusion and lighting of all the elements in the scene to understand which blurry colors to show. You can't just cast a few rays and stretch them to fill the periphery. Otherwise, slight camera or object movements might lead to chaotic flashing of peripheral colors as the reduced samples stochastically observe different scene elements while ignoring the rest.

I know of things like multi-resolution texture pyramids which can precompute a blurred version to allow undersampled texture rendering. And I know many games switch between different low/high polygonal representations of objects (or even cull objects) based on distance to the camera, which often produces obvious "popping" artifacts as the user moves through the 3D world. Are there general scene modeling techniques that can work as smoothly as multi-resolution textures to allow rapid or simultaneous use of different levels of geometric detail in one rendering?

I think those popping artifacts would be much more distracting if tied to every eye movement rather than only with navigation across a 3D world. Crude, culling methods would be inadequate if they just drop peripheral objects that might otherwise produce the kind of peripheral image that would cause the user to change their gaze...

Why was this down-voted? S/he seems to be making valid comments.