Hacker News Reader

Comment by Havoc | Hacker News Reader

I'm a little surprised that we haven't seen something more AI driven yet. (yes yes buzzword I know)

i.e. slice it not just into scenes but also into objects and do bitrate on that level. i.e. Face and objects in foreground get more. It seems we now have pretty small models that can do that sort of stuff (see Apple & MS ones recently) so should be feasible at scale.

I'd imagine you can also train an LLM on patterns that encoders choke on...chequered patterns etc.

View on HN

> i.e. Face and objects in foreground get more.

Encoders have been doing that for decades at this point.

AI-related models are actively being researched for multiple applications in video encoding. From generating predicted frames, to generating fine details, choosing tools and parameters for each frame, narrowing the search space, etc.

I know they do all sorts of clever predictive sorcery but I meant specifically what this image shows:

https://deeplobe.ai/exploring-object-detection-applications-...

Literally getting closer to understanding the context of the scene and what matters to a human rather than understanding the pixels

We've experimented with that at SVT (Sweden's public service broadcaster). Or rather, the video team has, I'm just a web dev, so I don't know the details other than what's in this blog post:

From the eyes of the viewer: Attention-aware video compression for improved low bitrate video https://medium.com/the-svt-tech-blog/from-the-eyes-of-the-vi...

Had to create an account just to ask more about this! Interesting approach, but the link is from last year and only talks about a research project.

Did you ever test this on a larger scale? And do you know if it is only applicable to one encoder?

After he got kicked out of MPEG, Leonardo Chiariglione opened his own standards organization[0] called MPAI, which is focused primarily on... standardizing machine learning techniques for video coding tools.

I'm not sure if they've actually shipped anything yet, though.

[0] With blackjack, and hookers: https://blog.chiariglione.org/the-mpai-framework-licence-app...

You mean transformer/diffuser based when you are talking about AI in this context? Because there are a lot of ml papers out there for encoding video. Stadia (RIP), Parsecs, Epic (Unreal Pixel streaming), META, Apple and Nvidia doing amazing stuff for their streaming services/VR/AR plays.

Yeah. This essentially

https://deeplobe.ai/exploring-object-detection-applications-...

If the model can figure out the key objects in the scene on a bit more deeper level then that can drive what parts the encoder needs to spend more bits on

The algorithms exists, but it requires way too much processing power and you also need to have a model already downloaded.

Then we can have many 6-fingered characters and other hallucinations. It's all fiction anyways so what's one for a other? I hope it either gets great fast or phases out as a fad.

No not model to generate it. Model to steer the encoder and tell it what parts to invest more Bitrate into

That makes sense and shouldn't have been that difficult anyway. For the barbie glitter scene there's obviously more chroma 'texture' to apply variable bitrates. Flat colored scenes could continue using lower bitrates. Where more complex models could be useful is avoiding entropy coding of useless details etc (e.g. if those details were artifacts of processing in the first place).

This is supremely easy to avoid. I'm not even sure how you arrived at thinking this will be a concern.

I didn't mean this one thing specifically. I was thinking more along the lines of Xerox copiers 'hallucinating' (randomly altering) digits in reports.

Reply on news.ycombinator.com