Project Starline: Feel like you're there, together

Because optometrists are illegal in my country (here only medics can decide what glasses you can use), currently I don't have stereoscopic vision, although my brain CAN do it, if I had the correct images sent to my eyes somehow. (one of my eyes muscles is slightly shorter than the other side thus the images on that eye are shifted unless I had an optometrist design me glasses with a prism).

So when I watched Avatar, an actually well made 3D movie, it literally felt more real than reality, despite it being obvious fantasy with aliens, floating rocks and all that stuff.

EDIT: for those wondering, I am from Brazil, here medical professionals often sue the shit out of anyone offering any service remotely similar to an optometrist, they are quite aggressive about it, some attempted to make even discussion of the subject illegal. And when I was trying to get the prism and asked around my medics about it, one of them went really ballistic, I honestly thought the guy was going to punch me. I believe the reason for that is that for many medics, designing glasses is their only source of income, a guaranteed one, since here is ALSO illegal to buy glasses without a medic desining them for you first, even if the new glasses are supposed to be identical to the old ones!

You sit down and you forget that there's technology happening. The person is there, in front of you. I don't know how else to describe it.

The testimonials in the video aren't exaggerating compared to my experience.

I count at least 5 waves of 3D technology starting in 1851 with the Brewster Stereoscope. Each time there's a surge of popularity driven by the legitimately amazing initial experience. And each time people slowly stop caring. People were incredibly excited about Avatar, and many thought it would change the movie industry. But how many people now go out of the way to see something in 3D?

https://lookingglassfactory.com/8k

If it's as seamless as it looks in the video that would be truly novel and exciting.

The big commercial need it turns out isn't so much realism as it is flexibility to accommodate people dialing in from different systems - phones, laptops, different types of telepresence setups from small single room to big conference rooms or even telephone connections etc.

Many years ago, we were all wowed by the life-size realism and had people come into offices for it . Nowadays these meetings have lots of people crammed in across multiple screens dialing in as they please and all the better for everyone :-)

There are demos of binocular 3d conferencing done with a lenticular display (lookingglass), although those large displays are extortionately priced (1/4 of the size of this google one is $3000...) - keeping them out of the hands of most devs.

No doubt google are doing the same, but can afford these larger displays.

You can easily find examples (and research papers) by googling for the relevant terms. Google claiming they have invented a "new technology" just shits on all those folk who dont have the publicity/funding of google.

https://en.wikipedia.org/wiki/Pepper%27s_ghost

It seems more like an art project than a tech usable in the coming decade.

> project (he has a Phd Princeton

Why is this part relevant? Was the PhD in light field technology?

How does this relate to advertising and the necessary surveillance to support it.

Will this "product" be set up to phone home and "update" by default. Will the price be "free".

Commercial viability. How much would someone pay for this.

>It's actually good enough that my parents have their morning tea together practically almost the same way they used to when they were physically together.

I don't understand why this sort of persistant 'portal' isn't more popular. But 10 inch screen is so small! A 55 inch TV is what, $300? Put it portrait mode and that's a life size portrait. The feeling of presence when the person's head is the correct size is so strong! I know Facebook sells something for the TV but can it be left on all day like that?

I want it feel like one half of living room is a shared space, at all times. Like the couch is split in half by a portal. Then it just cycles through a list of friends and family with the same setup to find someone else watching Netflix or whatever, and you can casually join them just like a real shared living space.

I don't need 8k or 3D tv. Just a 55 or 65 inch TV on the wall at the end of the couch!

https://portal.facebook.com/products/portal-tv/

There's another portal model that's as big as a small mirror.

I started working fully remote two and half years ago. Company has a couple of meeting rooms fully set up with Zoom, which I and a few other remote people used to connect to with my laptop for standups together with the in-office people. That's not the case everywhere but quite classic.

The company also had a dedicated TV+camera+24/7 Zoom meeting setup in the open space itself, so that remote people could connect to and say hi anytime. It was surprisingly nice.

Then one day on a whim I got a wide-ish USB webcam plugged into my Xbox One†.

The difference in perception is insane: instead of having this wall of people faces, like you're on several phone calls at once, I got a portalesque window between the office and my place. Instead of connecting people, it connected places††. But it only works if both sides are sharing a space instead of a selfie angle, and I think the way sound was picked up by a fixed element in the room played a lot as well.

So I'm a bit partial about this Starline thing, as it seems to be a step forward in "definition" but also feels like a blown up selfie angle.

† There's no Zoom app on Xbox (which is really a shame) so I used Edge, for which Zoom on the web is more limited and barks on webaudio (which works! Zoom just support only Chrome so I had to dial in on a phone). Also, Xbox+Kinect truly had the potential to be something amazing beyond games. I believe the marketing was botched alright but it was also way too early for the audience.

†† Which ends up connecting people on a deeper level. At some point I caught myself intuitively wanting to hand over physical objects through the TV.

I remember when I used to be in a long distance relationship with my now wife. We used to have the laptops on Skype for hours just to have each other's company, even if we were studying or doing something else. It was very unique at the time.

I'm quite jealous cuz back then we didn't have Echo Show, it'd probably be used just like your parents use it.

Solving gigantic scale problems requires a completely different kind of innovation than what you can achieve by pushing the pinnacle of what's possible.

(NB: I work at Google, but this comment has nothing to do with my work.)

The issues with connection stability and latency are very real, but I don't know if it's reasonable to expect Google to fix it; the issues there are probably more political than technical.

edit: Also, I think they did mention using AI for noise cancellation while videoconferencing in the keynote today.

In India, ISPs already advertise low latency and high speeds for "PUBG Gaming". The market evolves to solve the needs of consumers. Advertising for low latency for gaming was unheard of in Indian ISP scene, just a few years ago.

So having this tech, would induce customers to get better hardware, wiring, ISPs etc, and would induce ISPs to provide better service.

And it's really a question of money. If you want to fix all those mundane problems you describe, literally every one can be addressed by renting a better pipe and buying and setting up better equipment.

This Google product is clearly designed for high-end offices that already have all of those things under control.

If I could get video chat that felt like real life, that'd be worth running some wires for.

The office use case is probably more realistic, but some other related products (Surface Hub, Jamboard) haven't become as ubiquitous as originally imagined.

1. Call for free from home/couch with a regular camera in the privacy of my home.

2. Go to some location and pay a third party to do a less private call, and have a better visual experience.

The most obvious choice would be number 1 in 99.9% of all cases. In theory, privacy can be increased by creating a sound-proofed environment for your call, but in practice, that would easily become very expensive.

I’d first go with an Internet cafe style booth booking. Book an hour slot and get your coffee included.

Setup cafes in major cities and I think people would use this - could imagine setting a meet with a friend in another country. Parents showing off their new babies. Etc.

It seems we're seeing a second coming of distributed private comms.

Most companies don't give decent webcams already, I doubt they'd consider paying thousands for such a system.

To speculate, here are some reasons why keeping it secret longer might be hard:

  - They're going to do wider testing within Google.
  - They're going to start bringing outside testers in to try it.
  - They're going to start working with manufacturers.
  - Some newspaper got wind of it and is about to publish a story. (I think this happened with driverless cars?)

Also, it's nice for the people working on it when they no longer have to keep what they do a secret.

Edit: although, in this case, the timing mostly has to do with Google I/O starting today.

GOOG 411 was free slave labor for voice sample Captcha is free OCR training first and now road sign detection and image training AMP so that google can capture the web flow in the name of improving performance

This will be data for AI models specializing in video features of humans:

"AI" generated realistic human expressions which can then be used to do so many things AI gait detection aI partial face reconstruction

These all already have further applications which are harmful and will lead to tie in with Google

And in none of the other cases above has google published their compiled dataset. They "own that"

Google is cancer

the business trips i've been on either involved the installation of equipment, or were an excuse for somebody with budget to burn it on travel and expensive food/alcohol. or both.

i think plenty of business travel will survive just fine.

> What are Zoom, Facebook or Apple waiting for?

This is already a thing?

Facebook has Portal, Apple has iPad, Amazon has Echo, all of whom support Zoom and other video conferencing apps. The portal even has a moving camera to keep you centered if you're moving in frame, and the iPad does the same thing with an ultra-wide lens and some post-processing.

As far as dedicated videoconferencing hardware is concerned, Starline seems pretty late to the game. Although, I'll admit the fancy 3D imaging features is pretty insane.

Personally, I find that--for most people--the idea that working remote shoves a lot of cost onto employees vs. commuting probably off-base. (With some exceptions for people living in small city apartments near offices.) But installing a room-sized videoconferencing setup at home even for people with decent-sized houses is pretty silly.

Nothing beats in-person. Nothing!

I hope not. Video chat can really never be the same thing as meeting people in person.

The only thing missing is the after meeting drinks and dinner, but there will inevitably be services to put us all in a restaurant/bar environment, pipe in some bar white noise, have food sent from a local restaurant, etc. for an "in-person" virtual happy hour...

Is "Starlink" going to be a Gmail or a Wave? Hard to say.

I don’t think you’re a curmudgeon at all. Circa 2008, Google was innovating in a lot of different areas that were very valuable to the average user. The ROI on these services has gone way down.

Before you laugh or be prude, porn content was what made VHS and BluRay succeed (or are these urban legends?) and they were pioneers in stuff like online video streaming.

The (New) Nintendo 3DS has head tracking, but it doesn't change your perspective into the view port, which gives a very dizzying effect when you deliberately test the feature.

I would imagine its possible to extrapolate perspective if they had an array (N > 2) of cameras.

This is super cool tech, and can't wait to see an array of these installed in the secret undisclosed board rooms for the illuminati.

[0] https://www.google.com/amp/s/techcrunch.com/2020/06/23/googl...

Also, microsoft did this a couple years ago, without fancy volumetric display, just face tracking on their expensive 8k(?) tv for meeting rooms, and it was a complete flop.

That’s a big deal for anything which requires hardware you wouldn’t otherwise own. Once you hear “custom-built hardware and highly specialized equipment” that sounds like something you really want a commitment to not just begrudgingly patch but to continue to seriously invest in the product.

And its inevitable discontinuation.

http://creativity.arch.tamu.edu/_common/FoVI3D_DeepDrive.pdf

The explanation of how it works begins on p. 13.

It seems to basically just have a huge number of very dense pixels, and in front it has some tiny lenses that direct each pixel at a different angle.

Presumably there's a maximum distance you can view from because the further you get away from the display, the smaller the angle between your eyes. I would guess it's no coincidence that Google has set it up as a booth with a back wall that would ensure you keep close enough:

https://storage.googleapis.com/gweb-uniblog-publish-prod/ima...

I believe there are two basic approaches.

The first is "lenticular lenses" -- the same technology used to make those shiny postcards that change depending on what angle you tilt them at.

Here's a site with an example GIF and kinda explains how it works: https://setosa.io/blog/2014/07/07/lenticular/

Normally, the lenticular lens is glued directly onto a printed sheet of paper, but you can instead glue it onto a _screen_, which lets you change the pixels underneath. There are a lot of challenges here; for example, the pixels are usually located a bit behind the glass, so the standard lenses used for printing won't focus perfectly on a screen.

There's also another, simpler approach called "parallax barrier". This is what the 3DS used. Instead of using lenses to bend light, it _blocks_ the light coming out of certain pixels at certain angles. It's basically just an opaque sheet with periodic slits (the 3DS uses an LCD, so that 3D can be turned off by becoming transparent). Before I found a video of someone making one themselves using an iPhone and transparency sheets, but I can't find it at the moment...

3DS parallax barrier: https://www.youtube.com/watch?v=D-LzRT7Bvc0

Diagram from Wikipedia article: https://en.wikipedia.org/wiki/Parallax_barrier#/media/File:P...

A big disadvantage of parallax-barrier is that you end up blocking a lot of the screen, making the image darker.

Supposedly you can combine the two approaches to reduce the darkness.

https://www.youtube.com/watch?v=yetX2C3AdCI

There are probably even more tricks involved, and this is all a lot easier-said-than-done.

Here's a project that uses an LCD to make a dynamic parallax-barrier, that increases brightness by skipping the barriers wherever they don't matter:

https://www.youtube.com/watch?v=YrRnURU1W3E

Tangent: There's also been some attempts to do _the reverse_, and use distorted lenses in front of a camera to recover a 3D scene using only a single camera:

https://dl.acm.org/doi/10.1145/3072959.3073589 (link has a SIGGRAPH talk recorded which shows example results)

Well, at least we know that they weren't all white men; because if they were, this writer would sure have let us know about this very irrelevant information.

There is so much more we can do in terms of quality and immersion that we're not doing simply because bandwidth and connectivity are so low-quality and overpriced at most of our leaf nodes in the USA.

From watching the video, Google's conferencing setup is creating a 3D rep of the people talking and adjusting rendered view based on where the participants are seated. This is blending AR with videoconferencing. It would be interesting to see how their conferencing system works with multiple-people on each side. I know the video had a mother and baby in one segment, however is the 3D rendering based on the eye-level of the main participants?

This implies, AFAIK, that it either uses lenticular lenses (which is the tech 3D-cards typically use), or a parallax barrier (screen tech from 3DS). There are a thus sectors from the screen to the viewer, and you need to have your head placed so that your one eye sees one sector, and the other eye sees another. What the reporter describes is when both her eyes end up in the same sector, which immediately makes the result 2D. Note that there might be more than two sectors, so that you can move further sideways and get a realistic view, but each eye must all the time be in a different sector. It can also use head tracking to achieve such correction of your view wrt. movement of your head, since it evidently constructs a full 3D scene of you and the other side, it can render that from any angle.

Track both eyes, and then project an image to each eye based on its image in the room. The part I don't really understand is how it's possible to target the image to each eye. Maybe we have displays now which are like the 3DS screen, but with variable focal locations?

I was ok with Gmail scanning my emails to finetune targeted advertisement. But they don't do that, which is a nice surprise.

https://support.google.com/meet/answer/9852160?hl=en-GB#zipp...

Why would this be any different?

Especially, if this is pre-recorded and only sent for viewing (so no need too much bandwidth on upload side). I guess it would come down to the hardware and how much it costs.

So yeah...

https://blog.tonari.no/how-tonari-uses-tonari

Quite a bit of the technology used there seems destined to get more affordable as it's getting more widely adopted.

So, 12 years (back then) after the iPhone, it's reached a lot more than 5% of the world.

People are already depressed as hell from the lack of touch, real family, community and friends in modern society.

That said, of course this tech could have its uses, but mediated by the largest corporations on Earth that collect, sell and mediate everything about you and your friends? No thanks!

This is all made to evoke sense of closeness.