Sean-Der

Hopefully you are in a network that allows P2P! Then STUN just works and you can use any of the public servers (CloudFlare, Google, Twilio...)

Running your own TURN server would be trivial also. I have been tempted for a long time to make a 'TURN in a Box' that does autoconfig so people can run it easily on Hetzner/AWS

You can already do this! I saw this on X[0]. You can do WebRTC to Realtime API + getDisplayMedia.

[0] https://www.loom.com/share/22a165508ae5491dbd536fbbc5348fcc

WebRTC is running on the esp32! Library is libpeer[0]

[0] https://github.com/sepfy/libpeer

1.) Not something I did. I only worked on the software.

4.) Contracted because I wrote the Open Source libraries it uses. I created https://github.com/pion/webrtc and https://webrtcforthecurious.com the company reached out to me because https://github.com/sepfy/libpeer didn't work with Pion. Turned out to be a libpeer bug (which I fixed)

> were you really hooked

I have been doing hardware + WebRTC with lots of different projects since then. https://www.youtube.com/@seandubois86 unpaid, that's my proof at least :)

> Hooked on the technology stack.

Yes I enjoy doing WebRTC + C https://github.com/awslabs/amazon-kinesis-video-streams-webr... (one of original authors) so I enjoy the tech a lot.

> Hooked talking to Santa

I enjoy the unexpected behaviors of Voice AI at this moment in time. In the video I asked Santa about pi, and it responded in a way I didn't expect. I find it absurd/entertaining.

> Do you have kids that you built it for

Yes. I have a few toys/personalities that I built for my kid. They make up funny scenarios/personalities and I change the prompt and put it in a different toy. It's fun.

I would love to do that.

What I really wanted to build was a 'tour guide'. I could walk up to a piece of art in the museum and get more info on it. It would also be multilingual. At my local museum all the art descriptions are English only.

Might be too disruptive for a museum though. I want to discourage screen use/let people continue to use their eyes when learning.

It definitely takes coaching to use it. My daughter will interrupt and waits for Santa to repeat himself (and it doesn’t)

As technology gets better I’m excited for turn based voice ai to go away :)

This wasn’t built to be a general purpose phone. It’s a single purpose tool, if it had full SIP support it would need more expensive hardware. When I worked on this I knew I wasn’t building some interoperable general tool.

I am being honest with you. For me the ‘hacker spirit’ means cracking things open and learning how they work. So I totally encourage others to do it.

My experience with D&D was on the computer. I didn’t have friends that would play with me. I didn’t feel constrained by having computer driving the story. Games like Baldurs Gate pulled away from unhappy things happening otherwise.

id software had a profit motive right? As a kid it didn’t occur to me. I just nagged my parents to pay for Doom/Heretic.

I also have done everything to encourage/empower DIY. My hope is that users that are curious can learn more/build it themselves.

> Normally once a toy is bought, infinite time can be spent with it with no further financial transaction

I can’t think of any case where that is true. Books/toys all get worn and may need to be replaced. I have bought my son the same toy forklift three times because it breaks and he really loves it.

> Is it desirable to build a world where kids spend more time with this category of toy over others

I would rather see my kids play with this technology than consumption only (videos). Other play is better then doing Santa role play, but this isn’t close to be worse at all.

lol I love that.

My kids would find it so funny hearing Santa doubling down on bringing the wrong toy.

My hope is that the toy gets people interested/exposure to the idea.

Then the curious will open up the device/try to DIY. Lots of voice ai providers and microcontroller media code is open source!

Why? It’s probably just speech to text failing.

It reminds me of using early Windows. The unexpected behaviors of the software was what was fun.

Maybe not something everyone finds enjoyable.

How is roleplay with this object different then other toys? If you get lost in a D&D game is that bad because the world isn't real? Getting lost in Myst and making Doom WADs was a joy I have always been trying to recapture. I am constantly looking for a way to do that for others.

What do you think of my take here https://news.ycombinator.com/item?id=45575175? These 'LLM Role play' toys have hit a real fun spot with my kids.

Why?

I have had quite a bit of fun/bonding with my child over it https://news.ycombinator.com/item?id=45575175

The answer isn't no.

You can open up the phone and modify the ESP32. I do that pretty often with IoT devices. It's not as easy as setting a URL, but totally possible if you are determined enough.

Vapicon or Pion examples?

To be safe I would contact city zoning about the construction of your future backyard water park. Always good to start early :)

I didn’t do any of that, I just did media code!

I worked as contractor for company that has relationships with Walmart.

My 5 (at the time 4) year old always understood. We made a game out of it of ‘making new toys’ and she would tell me what it should say.

I would cut open toys and shove microcontrollers in them.

I think if you lie and tell a kid it’s a real person it would be damaging. My kid has fun role playing, she really suspends disbelief. When done she thinks it’s funny though/not confused.

Build it :)

It's the most fun I have had in a long time. Building a character and having it sit on your desk and chatter/say things you don't expect.

I love absurdism humor. This hits the spot for me.

It's already happening! I went to https://vapi.ai/vapicon and the stuff that people are already building is bonkers.

It would be naive to think that this technology would only be used for good. I have been working on Pion WebRTC for years though and have see lots of stuff getting built that didn't feel great. Not sure what I can do though.

Inference is always getting cheaper, so my hope is that restrictions like this can go away in the future.

I totally understand giving it a 'B'. But I promise you that I came at this project with sincere hope that I can build something that brought more joy then it cost into this world.

I have it at home and I think it's worth the money. My 5 year old uses it and the recordings I got from it I will keep forever.

* Santa tried to end the conversation and she said 'no no no wait, one more present'

* It thought she wanted a llama instead of something else and she hysterically laughed. As she gets older I don't hear that as much, and it made me so happy.

I call bullshit on things all the time, so I get the cynicism. But give it a shot! Seeing kids role-play with LLMs and especially when they hallucinate is a lot of fun. Honestly as the software gets better I think it might not be as fun. It almost feels like the joy of using Linux during the editing your Xorg config days. The chaos is what you fall in love with?

dunno

If I was building it to be 'local only' I would run the inference on a remote host in my house.

Having a microcontroller in the phone is nice because it is WAY less likely to break. I love being able to flash a simple firmware/change things would fighting it too much.

Oh! Also I do all the 'WebRTC/AI dev' in the browser. When I get it working how I like, then do I switch over to doing the microcontroller stuff.

I would DIY it, I think you will have a lot more fun/enjoy the end result more https://news.ycombinator.com/item?id=45575106

If you do I would love to help!

I would buy a dev board + build it yourself, you will get a much better experience then trying to reuse the existing thing.

I have written implementations target at specific boards. So go and buy one of these and boom stick it in anything you want. I have done this for my kids and have a bunch of different characters. My favorite is my daughter has a toy that pretends to be 'the ocean' it is so funny and existential.

* https://github.com/Sean-Der/realtimeai-embedded-respeaker-li...

* https://github.com/Sean-Der/realtimeai-embedded-esp32-s3-box...

I really loved the Sonatino[0], but can't get it anymore :(

If you start building something shoot me an email and would love to help! I want to unblock/enable this space so bad, I think these kinds of projects are just so delightful :)

[0] https://sonatino.com

Sorry about that, I reposted with a better link https://news.ycombinator.com/item?id=45562597

Sorry :( this link should be better https://news.ycombinator.com/item?id=45562597

The first time I talked with AI santa[0] and it responded with a joke I was HOOKED. The fun/nonsense doesn't click until you try it yourself.

What's even more exciting is YOU CAN BUILD IT YOURSELF.

* libpeer [1]

* pion [2]

Then go do all your fun logic in your Pion server. Connect to any Voice AI provider, or roll your own via Open Source. Anything is possible. If you have questions or hit any roadblocks I would love to help you. I have lots of hardware snippets on my GitHub

[0] https://www.youtube.com/watch?v=0z7QJxZWFQg

[1] https://github.com/sepfy/libpeer

[2] https://github.com/pion/webrtc

[3] https://github.com/sean-der

I think you would need the ESP32 to connect to another host. Doing Speech-to-Text, LLM, Text-to-speech is pretty intensive. Even if you connect to a Raspberry Pi.

But totally possible! It's a great idea and would love too help you build it :)

Wire some Open Source together and just start with a small collection of ogg files.

Nice job on this :)

I think it’s really fun how it’s all the same underlying protocols. Makes it easy to bridge these two protocols you would assume are dramatically different!

That's gonna take a long time to happen! So much hardware is WebRTC only. Even if the services migrate.

But anything is possible I won't bet against anything