Sean-Der

Thank you :)

When are you coming back to the WebRTC space, lots more cool stuff you could b doing :) I really loved [0] it's so cool that a user can access a server behind a firewall/NAT without setting up a VPN or having SSH constantly listening.

[0] https://github.com/maxmcd/webtty

Yes we are :) When OBS merges the PR [0] things are going to get very interesting.

Before you needed to run expensive transcoding jobs to be able to support heterogenous clients. Once we get Simulcast the only cost will be bandwidth.

With Hetzner I am paying $1 a TB. With AV1 or H265 + Simulcast I am getting 4K for hundreds of users on just a single server.

We will have some growing pains, but I am not giving up until I can make this accessible to everyone.

[0] https://github.com/obsproject/obs-studio/pull/10885

Those are issues in multiple implementations though! Lots of them are just issues in Chromium around Javascript (webrtc code wasn't even started yet)

That would be like saying saying 'webrtc is more secure then http' by posting this https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=http

IP Leaking has been fixed since 2019[0]. ICE/P2P is still a huge attack surface though. I have seen lots of other tricks being tried.

[0] https://www.youtube.com/watch?v=SqcW8kJAMJg

Currently only the sending part!

WHIP is 'pushing media via WebRTC' - https://www.ietf.org/archive/id/draft-ietf-wish-whip-01.html

WHEP is 'pulling media via WebRTC' - https://github.com/wish-wg/webrtc-http-egress-protocol/blob/...

WHEP isn't standardized/still changing a bit. After it lands I will try and get it into OBS/FFmpeg (and more)

How would you like to see/use the SCTP parts? I am not sure how to expose them since the WHIP IETF draft makes no mention/suggestion of it.

Most 'WHIP Providers' also support DataChannel. But it isn't a standardized thing yet

What security flaws?

This implementation is very small. I feel 100% confident we are giving users the best thing possible.

It's so exciting.

Especially with Simulcast it will make it SO cheap/easy for people.

I made https://github.com/Glimesh/broadcast-box in a hope to make self-hosting + WebRTC a lot easier :)

I am so incredibly excited for WebRTC broadcasting. I wrote up some reasons in the Broadcast Box[0] README and the OBS PR [1]

Now that GStreamer, OBS and FFmpeg all have WHIP support we finally have a ubiquitous protocol for video broadcasting for all platforms (Mobile, Web, Embedded, Broadcasting Software etc...)

I have been working on Open Source + WebRTC Broadcasting for years now. This is a huge milestone :)

[0] https://github.com/Glimesh/broadcast-box?tab=readme-ov-file#...

[1] https://github.com/obsproject/obs-studio/pull/7926

I am so incredibly excited for WebRTC broadcasting. I wrote up some reasons in the Broadcast Box[0] README.

Now that GStreamer, OBS and FFmpeg all have WHIP support we finally have a ubiquitous protocol for video broadcasting for all platforms (Mobile, Web, Embedded, Broadcasting Software etc...)

[0] https://github.com/Glimesh/broadcast-box?tab=readme-ov-file#...

Lots of cool stuff getting built with this already!

On X someone was posting little vehicles/toys that were being driven around. They were being driven by a speech input that was processed by some AI and that caused hardware functions to be fired.

Fun times in WebRTC :)

Fantastic project, and the team behind it is really good! The developers I have worked with are passionate about building things the right way (not just making it work/adding kludge).

I felt like I was seeing the future when I saw the visualization/rendering of PeerConnection stats on the server side. The video compositor is really neat also how they have it working with live modifications.

I wish I had more time/a chance to use it on a project myself.

Did you use the WebRTC/WHIP ingest for this?

sean @ pion.ly please email me any time.

Offer is open for anyone. If you need help with WebRTC/Realtime API/Embedded I am here to help. I have an open meeting link on my website.

My plan is that Espressif’s WebRTC code[0] will hook up to pipe at [1] that gets you the freedom to do whatever you want.

The design of OpenAI + WebRTC was to lean on WebRTC as much as possible to make it easier for users.

[0] https://github.com/espressif/esp-webrtc-solution

[1] https://github.com/pipecat-ai/pipecat

I hope these toys could be a joy/comfort for kids that don’t have a parent that cares.

I poured hours into games/programming because it was a happy place away from school etc… These toys could be the same.

This technology is neutral, but I see so much potential for projects that do good.

I think it will always feel unnatural as long as 'AI Speech' is turn based. Right now developers used Voice Activity Detection to detect when the user has stopped talking.

What would be REALLY cool is if we had something that would interrupt you during conversation like talking with a real human.

I get a `Request has expired` could you upload somewhere else?

This is wonderful, really great job on this! For me physical devices is when it really starts to feel magical. My pre-schooler never engaged with Speech-to-Speech examples I showed her on a screen. However, when I showed her a reindeer toy[1] on my desk that tells joke that is when it became real. It is the same joy/wonder I felt playing Myst for the first time.

----

If anyone is trying to build physical devices with Realtime API I would love to help. I work at OpenAI on Realtime API and worked on [0] (was upstreamed) and I really believe in this space. I want to see this all built with Open/Interoperable standards so we don't have vendor lock-in and developers can build the best thing possible :)

[0] https://github.com/openai/openai-realtime-embedded

[1] https://youtu.be/14leJ1fg4Pw?t=804

You have multiple implementations (Go, Python/Typescript) and servers (Janus/Jitsi)

I was afraid if I pushed any software in particular it would dissuade other groups from using it

I’m glad you had fun with WebRTC. It’s super cool tech.

If you are ever back in WebRTC land would loved to help https://pion.ly/discord

The book is vendor agnostic. I wanted it to be timeless and vendor agnostic.

The lack of code made all publishers I approached reject it :( I would love to see a hard copy someday.

I had hoped if I put no code in it the WebRTC community would feel more comfortable with it. If I made it Pion specific, would greatly reduce its reach.

Depends!

If you want unordered/non-sequential it could help.

If you don’t care about those things could be more work for no value.

Why do you consider it the status quo? It isn’t used in servers, embedded devices or software like OBS. Yes it’s popular because of usage in Chromium/Mobile, but I don’t believe that qualifies it as status quo.

aiortc is good enough to empower lots of businesses/projects! If anything comes up that really is a show stopper it will get addressed.

Of course!

I think the 'What, Why and How' captures the purpose and joy of learning. Nothing is better then getting lost in a fun problem. I don't want to just understand how it is solved, but the story about how it got solved.

If you have any feedback on how the book could be better would love to hear!

Of course! Glad you enjoyed it. If you have any feedback on how to make it better I would love to hear :)

WebRTC can create a NAT Mapping via STUN (UDP). https://webrtcforthecurious.com/docs/03-connecting/#nat-mapp... that is pretty powerful compared to Client/Server HTTP.

Dependent on the firewall, but most I have seen allow NAT mapping with different behaviors.

I am so happy that WebRTC has so many implementations. Every developer thinks differently, it's good to have a implementation that matches your mindset.

* https://github.com/shinyoshiaki/werift-webrtc (Typescript)

* https://github.com/pion/webrtc (Golang)

* https://github.com/webrtc-rs/webrtc (Rust)

* https://github.com/algesten/str0m (Rust)

* hhttps://github.com/sepfy/libpeer (C/Embedded)

* https://webrtc.googlesource.com/src/ (C++)

* https://github.com/sipsorcery-org/sipsorcery (C#)

* https://github.com/paullouisageneau/libdatachannel (C++)

* https://github.com/elixir-webrtc (Elixir)

* https://github.com/aiortc/aiortc (Python)

* GStreamer’s webrtcbin (C)

See https://github.com/sipsorcery/webrtc-echoes for examples of some running against each other.

I need to finish it but 'Reading the SDP' I hope to solve with https://pe.pion.ly/ (https://github.com/pion/explainer). You can just your Offer/Answer in and interactively explore what has been configured.

WebRTC's complexity can be frustrating. I believe it is inherent to how many things it is trying to solve. If an alternative arrives that solves everything WebRTC does, it will end up being just as complex.

Or maybe not! Time will tell.

Yep! ICE lets you try a bunch of different ports and protocols.

So you can have a remote peer and try to contact it via UDP/TCP/TLS. You can even attempt via multiple interfaces (Wifi and 5G).

You can then measure the packet loss/latency across these different paths and figure out which one is best.

I hope everyone enjoys WebRTC for the Curious (I am one of the authors). If you have any ideas on how to make it better I would love to hear :)

WebRTC is absolutely magical. Having 'Web' in the name does it a disservice. I see it used to.

* Remotely control excavators

* Security Cameras

* Connect autonomous robots in factories

* Low latency streaming from sail boats and rocket ships

* And all the use cases you expect (Conferencing, Telehealth etc..)

What is the latest/greatest when it comes to voice models? Orpheus is what people have the most success with?

I only concentrate on the WebRTC side, not aware of all the offerings.