Panasonic’s Innovative Augmented-Reality HUD Could Be in Cars by 2024

> If it notices that the driver's head has turned, it adjusts the HUD elements being projected in less than 300 milliseconds to compensate.

It is very slow.

People feel discomfort when different "overlays" lag by as little as few tens of milliseconds.

People feel uncomfortable working with touchscreens because they feel "unnatural" when they push the button, and it only reacts, and plays animation half a second later.

What people say what feels most "fake" about touchscreens is stuttering, and lagginess.

I believe you really need to go to sub-millisecond levels of latency for it to feel "glued" to the real world.

I had a touchscreen project in the musical instrument space, with a pressure sensitive touchscreen, where button animation had to match the touchscreen pressure.

Even with all tricks, lag on Android was making it completely unusable, not to say Android's audio API is completely not suited for serious audio work. In the end we went bare metal with it.

The lowest latency we got was 18ms by using the touchscreen interrupt to set the sensor value as a pointer to bitmap image of a button animation frame.

Yet, some more experienced musicians said that it was still "felt off."

I'm thinking to go full FPGA next time I need to do any latency sensitive HMI with kinetic-screen interaction.

Hey folks, Phiar founder here (YC S18), we are the AI and AR tech behind this Panasonic AR HUD.

Just wanted to help clarify that 300ms means "delay", not refresh rate (FPS). We run our entire AI and AR at under 33 ms per frame, and we can get to about 60 FPS of optimized throughput. The 300ms of delay mostly comes from the hardware display process: it takes time for the camera to receives an image, and passes that image through many different pieces before getting to the processor that we can use, and then it takes again some time to be rendered. With that said 300ms is the upper-bound and typically the delay is less than that. Further, you may not have noticed but there's similar delays for iPhone AR games, as well as to all other AR headsets, and that delay is there just for the videos to display (no AR, no AI).

We also recognize that distraction is a risk, and therefore UI/UX design is highly important here. Ultimately, we believe that overlaying on a live view in a clean and effective way is better than people reading directions off of a synthetic 2D map on a different screen that takes your eyes completely away from the frontal view.

Definitely appreciate all the feedback and discussions though, we will continue to improve and do our best to deliver a product that people want!

- BTW we are totally hiring: software engineers, PMs, sales, AI/AR scientists, and more!

Oh, no. Screen clutter for cars. You just know ads are coming. Also, 300ms of lag is going to be painful to watch.

Watch "Hyperreality", if you haven't seen it, for where this leads.

[1] https://vimeo.com/166807261

I am the only one who wants interface to the car comparable to the race cars? More knobs and dials instead of fancy touchscreens! Unless the car is really self driving and I don’t need to look at the road. 99,9% of the time I drive on familiar routes. On the highway I need a device that shows traffic jams ahead, not a fancy navigation.

Edit: typos.

Notice the bright yellow icon representing the bicycle. The risk is if some bicycles are not recognized (not showing icon), people who depend on the icons might be oblivious that there are bikes there.

Like when I write email and some parts in bold, people tend to read only the bold ones.

> If it notices that the driver's head has turned, it adjusts the HUD elements being projected in less than 300 milliseconds to compensate.

Assuming the road isn't just a flat straight road all the time, there will be uphill and downhill drives. Turns, curves, and twists. All of which affect your FOV. The YouTube trailer didn't show the actual windshield, but a view recorded from a mounted camera.

I wonder how this will compare to a bad FPS gaming experience.

I've been saying it forever since I got a HUD in my car.

This is the way. HUDs are an absurdly awesome coolness and safety feature. I don't care about how the dashboard looks because I don't want to look away from the road! Seeing this is extremely impressive and I hope that we can get AR technology to see widespread adoption in cars ASAP. The fact that we have to wait until 2024 (and for us Toyota or Lexus drivers 2028-2030) is just embarrassing. Why isn't this coming in the next year?

Tesla's refuse to add them and the aftermarket options suck, so I can't buy one either which is extremely sad.

Maybe an unpopular opinion: I don't want this. I don't want any of this.

Don't block my windshield with your "helpful" alerts. I don't care, I can drive without them, that's how I (and millions of others) learned to drive.

I want a physical key.

I want the engine not to cut off when I'm stopped at a traffic light or stop sign.

I don't want more cameras, whether facing the occupants, or facing the road.

Cars are already "good enough", and have been for years. Why do we need to keep adding software and other janky "features" that create more failure points and more expensive repairs?

Can't wait enough for HUDs to become mainstream for cycling too. Current offerings are way too expensive and immature. The way I envision these should go is helmet mounted visor with a replaceable, helmet mounted battery; standard lenses you can source from various vendors; swappable HUD modules; manufacturers could offer complete package (helmet, computer/projector, lens, battery) lineups and employ their regular pricing strategies. Not sure why nobody came up with such a system yet, all technologies are already there.

i have missed many turns where the phone says “in 500 feet go right” and i have no idea how long 500 feet is and there’s many places to go right. it would be so nice to have a line drawn on the road

Have always wanted HUD for Maps! Looking away from the road can be very dangerous, even if the phone is mounted closely to the wind shield.

The first time I saw HUD navigation was when playing Need For Speed 9. It'd be awesome if that becomes the reality.

I would like to have a simple an clean HUD. No flashy warnings. And I would like to have a lot of knobs and dials. Not a friend of touch screens in a car at all.

This would be interesting in busy environments or when the driver is less alert. Humans can only track about 4 objects at best (unless you are really lucky and can do 5) so instead of not seeing an incoming pedestrian or cyclist a big bright orange label would be way better since something else has already done the filtering for you.

300ms is a joke. If 300ms is the best you can do then reduce the amount of information you put up.

Maybe only put up satnav directions. Like “take next turn right”. 3fps is enough for that.

But if you’re gonna draw lines and highlight cars and bicycles at 3 FPS you’re just gonna make everyone vomit in their own car.

I would put a high quality HUD in my weekend car in an instant, if I could find one. I love the way they can communicate information efficiently. Still haven't found anything I would remotely call acceptable and available though.

How do they deal with projection correction?

I mean unless the HUD is in a helmet, then things like head position and driver's height will create rather large mismatches between what is the intended projection and actual road features, no?

What I want is something like Tesla's underbody radar for every vehicle, makes overtaking extremely safe.

That seems soon, but not soon enough at the rate self-driving tech is advancing, we may not need/use it.

I wonder how well it performs at night.

Anyone know if this project is hiring software engineers in America?