Sean-Der

* Use FFMPEG lib directly * GStreamer * Use libvpx, examples is in mediadevices repo

It's really cool how you can have a heavy server doing all this cool computer vision stuff, but then a simple client that just does WebRTC.

If you adjusted the example you can quickly be sending OBS or ffmpeg in. Then that gets really cool/powerful

The good detection ones required users to download something first, so I was worried the friction would make people less likely to use them. The one I almost went with was hand-gestures, but it wasn't as reliable.

I haven't used opencv that much. I see people talk about in Pion discord (and examples like this) and gets me into. I think someone more proficient could build a more compelling demo pretty quickly :)

[0] https://gocv.io/writing-code/more-examples/

Pion is a open source collection of Go software that lets you build real-time media applications. No corporate entity behind it, just developers enjoying their craft! Lots of opportunities if you are looking to learn or want to build something interesting but can't find the right problem.

Outside of OpenAI lots of mechanisms exist stuff like STIR/SHAKEN[0]

[0] https://en.wikipedia.org/wiki/STIR/SHAKEN

But for now yea waiting for Webhook is way to do it.

I am curious about adoption and features that would make big difference to users :)

Once I got it working it felt really cool though. As a user I don't want to configure WiFi on the microcontroller at all. I would be really cool if I could walk up to a 'smart device' and set my phone next to it and do zero configuration.

-----

I don't expect you to know anything about me. It made me feel like you have written me off/dismissed me when you mention HomeAssistant + WebRTC. HomeAssistant uses Go2RTC and the WebRTC library it uses is Pion[0]. I created that and maintain it. Getting WebRTC on tiny devices is something I have been working on for years and always doing it Open Source/making it best for users.

-----

> comes off as doing things that have already been done before but presenting as if something novel was done.

I don't think 'Hardware AI Assistant' is a novel idea. What I hoped was a novel idea was putting it in an easy to use package for others to build upon and use. WebRTC + Hardware is something I have been trying to make easier for a long time https://github.com/awslabs/amazon-kinesis-video-streams-webr... [1] I wrote this code, but things still felt too difficult.

When ESP32s got powerful enough to do WebRTC I wrote [2]. Reflect inherits from that. So I am proud of that. I have been on this journey to make RTC + cheap hardware possible so that others can build upon that.

-----

Again I really appreciate your comment, and sorry to be so defensive. Someone I really respected (and I thought they respected me) said the same thing about my work not being novel. They said people have been building security cameras for years that use WebRTC, you are over inflating what you are doing. That has stuck with me. So part of me does fear that I am wasting my time trying to solve these problems.

I don't think what I am doing is novel. I do think that I am solving it differently because I make it accessible/Open Source. Most people solving these problems/building it just keep their code at work and don't try to help others use it.

If you are up for it shoot me an email sean@pion.ly and https://www.linkedin.com/in/sean-dubois/ I would love to have a friend that calls me out/is honest whats good work and what is just BS :)

-----

[0] https://github.com/pion/webrtc

[1] https://github.com/awslabs/amazon-kinesis-video-streams-webr...

[2] https://github.com/Sean-Der/embedded-sdk

If I put these devices through out my house it would allow me to switch AI personalities by proximity.

You can also use the device without your phone. These devices are also very cheap. I think you could do audio only for around ~5$

Also I was hoping to push people toward a RTOS. Better experience then a raspberry pi, I can cycle power and be back way quicker. Also cheaper/more power efficient.

I also think I unfairly like ESP because it’s an excuse to write C :)

https://m.youtube.com/watch?v=HbO18Elw9WY

Are two that I know of. Try it out, if you hit any roadblocks @ me on pipecat discord and would love to help

You can use it to build lots of different real-time communication projects. Conferencing, Send your audio/video to GPU servers for AI, broadcasting and lots more.

It’s a super fun space to be in

I was tempted to put Erik Satie in the README video. Didn’t want to risk copyright issues

My primary work is on backend WebRTC servers. This was just an outlet/fun side thing to do client and embedded work. I love writing C and do microcontrollers. I just can’t seem to find a way to do it full time:(

I see so much potential if I can make hardware hacking + WebRTC easy. Not just for AI assistants but security cameras + robotics. If anyone has questions/ideas/feedback here to help :)

You can see the code for it here https://github.com/paullouisageneau/libdatachannel/blob/mast...

If anyone is doing things with WebRTC broadcasting (or thinking of trying it out) would love to hear :)

* Pipecat lets you switch between different AI backends with just a little code

* Espressif is so easy to hack on (and cheap).

Getting closer to this being easy enough for more cool projects to take off :)

[0] https://youtu.be/6f0sUEUuruw

Thanks for publicizing you used neko. I am sure it means a lot to the authors :)

I love what you have done with utsuru. Would you be up for sending a PR to [0] to deprecate it and point to your repo?

If you are up for it you should join the Discord in https://github.com/glimesh/broadcast-box and maybe https://pion.ly/discord I would feel honored to have you in these communities. I think what you are doing is super innovative, and I would love for other people to see it :)

Lastly have you posted this on X or LinkedIn? Could I share with others. I am constantly trying to show people 'this is the future of video' and you are making it happen.

[0] https://github.com/Sean-Der/obs-into-discord

https://github.com/pion/webrtc/tree/master/examples/save-to-... here is an example of a server that does what you need on the backend

We could go make a better/simpler standard for video streaming that is TCP. What a giant battle though that would never seen adoption .

I have accepted/embraced the challenge of making WebRTC as accessible as possible. Stuff like WebRTC for the Curious in hopes of making it less painful for everyone dealing with the complexity :)

You could also join the Pion one https://pion.ly/discord

Other place is video-dev Slack https://www.video-dev.org/

X also has a great community of hackers!

webrtcHacks has an article on it[0] kind of old, but captures the spirit of it!

[0] https://webrtchacks.com/tag/simulcast/

ICE (protocol for networking) supports this today. It just needs to get into the software.

With bitwhip[0] I got it way lower then that even.

[0] https://github.com/bitwhip/bitwhip