the_king

The single core performance looks really fast.

  Chip | Geekbench Score (Process)  
  ---- | ------------------------  
  M1   | 2,419 (5nm)  
  M2   | 2,658 (5nm)  
  M3   | 3,076 (3nm)  
  M4*  | 3,810 (3nm)

In my experience, single-core CPU is the best all-around indicator of how "fast" a machine feels. I feel like Apple kind of buried this in their press release.

M4 benchmark source: https://browser.geekbench.com/v6/cpu/8171874

Peter Thiel was widely criticized this spring when he said that AI "seems much worse for the math people than the word people."

So far, that seems to be right. The only thing o1 is worse at is writing.

This is really cool. Curious to see more about how the database can be queried. I don't write much SQL these days, and I have no dedication to Postgres, but it does integrate with pretty much everything. Also curious how I'd go about the basics in Instant.

For example, creating a user table and ensuring that emails are unique - I've done it 50 times with Postgres. Is that part built out yet?

Very cool. Appreciate the "Don't Make Me Think" API.

(written with aqua)

It makes sense for desktops but not for devices with batteries. I think Apple should introduce a new device for $5-10k that has 400GB of VRAM that all Macs on the network use for ML.

If you're on battery, you don't want to do LLM inference on a laptop. Hell, you don't really want to do transcription inference for that long - but would be nice not to have to send it to a data center.

I'm all for running as much on the edge as possible, but we're not even close to being able to do real-time inference on Frontier models on Macs or iPads, and that's just for vanilla LLM chatbots. Low-precision Llama 3-8b is awesome, but it isn't a Claude 3 replacer, totally drains my battery, and is slow (M1 Max).

Multimodal agent setups are going to be data center/home-lab only for at least the next five years.

Apple isn't about to put 80GB on VRAM in an iPad for about 15 reasons.

I would use Pytorch as your starting point. Its metal backend is pretty quick on Apple Silicon, and it's the most widely used library for everyone from hackers to foundation model builders.

Thank you very much. I can't say how much these comments motivate us.

When I was a kid I saw tech as a really aspirational thing (jobs, iPod, touch screens, Skyrim). What can be done with modern GPUs and transformers is making me feel like that again.

Thank you. 100% I'm with you on speed.

I think we will be in a much better place speed-wise in a few months; some of that will be our stack, and some of that is what is happening lower down in the stack, but it will be meaningfully faster and more responsive soon.

Yikes! I think we may have normal mode a bit too conservative at the moment. We've got a lot more tuning to do.

One thing you can try if Aqua doesn't seem to be "getting" what you're after is to say something like "make this the email," or "transform this into the email I want to write." Sometimes it needs an extra push.

Great idea.

This is definitely worth looking into. We don't want to slow down the pace of development, but this might be a case where the partnership makes too much sense.

A huge portion of what goes on in universities is writing.

edit: Reading this back, I thought I sounded too eager to partner with universities. We're a tech company, and quality and performance will always come first.

Great question. I just tested it, and Aqua was smart enough to figure out which "GPU" I was talking about using context.

In the example below I asked it to "make it H100 GPUs" and it only modified the GPU in the list.

Aqua isn't perfect though, and while I think we are mostly solving this case there are plently where we need to do better.

---

Hey Team,

I just had a chat with the marketing people, and they confirmed we should buy enough GPUs to meet the demand.

Our Equipment List:

- 1,000 H100 GPUs

- 1,000 processors

- 500 NVME SSDs

- 200 standard racks

edit: formatting

That's a great idea, we should do that.

I have a similar dream, we'll make it happen!

The programming use case is pretty interesting.

Appreciate it.

I think the preference for voice versus typing is something that hits everyone differently, and I think as the reliability and speed improves, more and more people will find themselves using voice as a "tool in the toolbox," which aside from the occasional "Hey Siri, set a timer," isn't the case today.

Thanks, appreciate it. We can do a lot better than most people experienced today in terms of latency.

Thanks! Sorry about the lag, today was slower than normal, but we need to improve latency overall.

Did you wait for the text to turn blue and then black? And were the twos still wrong then? The real-time text is non-final tokens and has many more errors than what is ultimately committed to the document (but committing is slower than we'd like at the moment).

Awesome. Agree on the copy-paste annoyance, we're working on more clients.

But I do think that the reliability needs to take a few more steps before it becomes a true keyboard replacer.

It's 1000-1500 words. I know that seems cheap of us but the cost to run the Aqua stack is eye-watering right now. We will increase this amount as we optimize.

Hope this can be helpful. We know there are still many kinks to iron out.

On another note, I think once you leave school dyslexia can become a wash or even a net positive in the right setting. I think whatever the brain config is can be a huge unlock for creative thinking - it's not always super helpful in the school context, but can be really asymmetric in tech and probably other industries.

You can use it in the browser right now! but we get... native is better for voice stuff and we'll be in more places soon.

We'll be releasing a custom dictionary and templates soon. We are testing them internally now, and they aren't quite reliable enough to release, but we understand how important this is for many workflows.

On accuracy, we benchmark very well against even large async models, with a WER of .05-.06 and when Aqua does make a mistake you can often correct it by just telling it "no it's our side not outside" and it won't mangle the text.

Sorry about that, will fix asap. I love firefox.

Fixed. Thanks for the heads up.

I think I developed "bad handwriting" partially to hide misspellings—this is necessary in school though.

On pure WER we are state-of-the-art in our testing, but more importantly, mistakes in Aqua are correctable.

So you can speak your mind instead of having to wait until you have the perfect sentence and then dictate it.

That said, we know it's not perfect, but we know a few more months of work will have it really solid.

Sorry about that. We know we need to be better about that and of course add more languages.

A few things to try to maximize your accuracy right now are:

- Don't use AirPods, especially not AirPods Pro. Most built-in laptop mics or EarPods or a gaming headset are perfect. It doesn't need to be podcast quality.

- Correct transcription mistakes as you would a person, then "plow through" and often the error will be corrected as you complete the sentence.

We use our own fusion model in the transcription pipeline for intent understanding from encoded audio, but most of the rewriting tasks like "Turn this into a list" call out to fine-tunes of GPT-4. It's a combination.

The fusion model is similar to the architecture described here: https://arxiv.org/abs/2310.13289

Thanks!

We had to make a bunch of breaking API changes over the last week and the Mac app isn't ready to go on it quite yet, but we'll bring it back as soon as we can, max two weeks, hopefully sooner.

Appreciate the feedback, we'll take a look at that.

We agree totally; voice only can be ridiculous, for example, if you're spelling out a username or something.

The sandbox doesn't have typing, but the full app does - you can switch between typing and talking seamlessly there.

(written with Aqua)