zammitjames

That’s a great point! While we do our best to simulate an identical case, if the agent responds differently, our focus is on whether the key evaluator or goal for that replay set passes or fails. We use that as the source of truth and flag the exact moment where the conversation diverges from the expected flow.

We help capture discrepancies between your transcription model and errors in order to effectively calculate a Word Error Rate (WER) as part of your evaluation process. Post-call transcription tends to be more accurate, and we’ve seen teams manually do this by hiring humans to label a dataset and test against it for WER calculations.

By providing a more accurate baseline, Roark helps teams quantify how well their production transcriptions match reality and flag cases where the model is introducing errors that could impact downstream agent performance. That way, you’re not just testing if your agent responds correctly, but whether it’s getting the right inputs in the first place.

Great question! Roark is named after Ted Roark from Chuck — both of us (Daniel and I) watched Chuck a lot growing up, and the name just stuck with us.

No deep meaning, just something we liked and thought sounded cool!

Thank you!

Great question! Most teams deploying voice AI do have testing and QA flows, but they’re often manual, brittle, or incomplete. Unlike traditional software, voice agents don’t have structured inputs and outputs — users phrase things unpredictably, talk over the bot, or express frustration in subtle ways.

Some engineering teams try to build internal testing frameworks, but it’s a massive effort - they have to log and store call data, build a replay system, define evaluation criteria, and continuously update it as the AI evolves. Most don’t want to spend engineering time reinventing the wheel when they could be improving their AI instead.

The teams that benefit most from Roark are the ones with strong QA processes — they already know how critical testing is, but they’re stuck with brittle, time-consuming, or incomplete workflows.

Great question! There’s been a lot of movement in this space, but most existing solutions focus on simulation-based testing—generating synthetic test cases or scripted evaluations.

Roark takes a different approach: we replay real production calls against updated AI logic, preserving actual user inputs, tone, and timing. This helps teams catch failures that scripted tests miss—especially in high-stakes industries like healthcare, legal, and finance, where accuracy and compliance matter.

Beyond replays, we provide rich analytics, sentiment & vocal cue detection, and automated evaluations, all based on audio—not just transcripts. This lets teams track frustration, long pauses, and interruptions that often signal deeper issues.

Would love to hear more about your assistant - how are you thinking about testing and iteration?

Appreciate the feedback! To clarify, Roark isn’t handling authentication itself - it’s a testing and observability tool to help teams catch when their AI fails to follow expected security protocols (like verifying identity before sharing sensitive info).

That said, totally fair point that this example could be clearer—we’ll keep that in mind for future demos. Thanks for calling it out!

Thanks, Jan! Would love to hear more about what you’re working on! Just sent you an email.

Appreciate the feedback! Completely agree - authentication should be handled at the system level, not just in prompts. This demo is meant to showcase how teams can build test cases from real failures and ensure fixes work before deployment. We’ll consider using a better example.

Thank you so much for your tip! Maybe I should have shared this instead: https://www.youtube.com/watch?v=1QDZJcnLATI

thank you!

Hi! Thanks for your feedback! We'll get right on it :)

Kwaver - Change the way music is born.

- Music Collaboration App - https://kwaver.com

We were part of the Early Access Program for the expanded Firebase and used it to build our music collaboration app Kwaver. With the new features, they did a nice job of collecting a bunch of related mobile products (Analytics, Push Notifications, Storage, Authentication, Database, Deep Linking, etc) into a pretty cohesive platform, and it's saved us a bunch of time.

With Firebase Analytics we can track events, segment audiences (according to user properties; active, dormant, inactive) and take action according to the user segment. We are able to send push notifications (also using Firebase) to dormant male users who play the piano for example. Another cool feature is Remote Config, which gives you the option to ship a number of separate experiences and track the user interaction. Like A/B Testing but way more flexible.

For us, the best product is the existing database product they had, as it really improves our user experience to ditch the 'pull to refresh' button' and have our app respond to changes live.

We have been waiting for Google to provide developers a more complete mobile solution for a while now, and they’ve done it superbly through Firebase!

Feedback; It would be really cool if Firebase could implement UTM codes to be able to track user acquisition and be able to automate actions according to User Properties.

Shameless plug: if you're a musician (or a music fan), we'd really appreciate if you could download our music collaboration app, try it out and give us feedback. It’s available for free on the app store; The following link will re-direct you there later today. http://kwaver.com