huac

it's difficult to gauge from outside / as a consumer, but what's interesting is rarely where models are at a given point in time, but rather where the model/team will be with similar amounts resources. it may very well still be Udio (who presumably have significantly more resources than Sonauto), but I would hesitate to say that a compute advantage counts as being 'far ahead.'

Another useful model to compare to would be DAC https://github.com/descriptinc/descript-audio-codec

This is the codec that TSAC extended, so it could be a nice comparison to see. I'd also echo Vocos (from sibling comment), it operates on the same Encodec tokens but generally has better reconstruction quality.

There's a number of scaled AMD deployments, including Lamini (https://www.lamini.ai/blog/lamini-amd-paving-the-road-to-gpu...) specifically for LLM's. There's also a number of HPC configurations, including the world's largest publicly disclosed supercomputer (Frontier) and Europe's largest supercomputer (LUMI) running on MI250x. Multiple teams have trained models on those HPC setups too.

Do you have any more evidence as to why these categorically don't work?

I concur

english speaking rate: ~150 words per minute

gpt tokens per english word: ~1.3 tokens per word

1M hours = ~12B tokens

I'd fudge that up a bit because it's probably more tokens from non-English but comparatively seems like not much

time to first token != tokens per second

remember that EU -> US is ~150ms unavoidable latency, for example. then your comparison is local H100 vs Grok + 150ms latency to first token.

why don't you stream the results?

> If I ask to deduce the gender of my voice, can it do that?

This iteration is not trained to do so. But the general model structure should work, i.e. if you finetune with instruction data to do so.

> Training a projection layer makes sense, but ultimately you'd want to output audio conditioned on the input rather than text. Is there a way to train a reverse projection with some kind of skip connections to take audio input into account? Or an end to end audio model?

Training an end-to-end model is the eventual goal, yeah; discrete tokens is the most pragmatic approach then. I've successfully trained an end-to-end model at small scale; am in process of getting more data (and money/compute) to scale that up.

ChatGPT voice takes the cascaded approach - Whisper to transcribe speech to text, then to GPT, then to TTS. We skip the transcription step.

Latency: OpenAI's implementation is quite slow - 5+ seconds to get a reply - but even optimized solutions are far slower than this approach. Fundamentally, most transcription models do audio -> audio encoder -> text decoder. By skipping the text decoder, we are always going to win on speed, we just do less work.

Emotion: transcription cannot transcribe emotion, only the text, but emotion - eg if the user is sarcastic, angry, happy, etc - plays a big role.

Transcription errors: in a cascaded approach, any mistake in an earlier step propagates to the latter steps. Mistranscription means that the LLM will make a mistake. But the LLM's are really really smart, certainly smarter than the transcription models - they'll get it right more often.

'narrow scenario,' perhaps, but one that also happens to closely match rumors for GPT4's size

VSCode is a good target for single-player editing, and I can see something like this being helpful, but what does a collaborative experience look like, eg if you have a team of folks all working on the same prompts?

the famous FB ads paper (from 10 years ago!) combines decision trees with a logistic regression and shows a significant improvement: https://research.facebook.com/publications/practical-lessons...

feel free to extend logistic regression to an MLP :)

yeah my code needs to use multiprocessing, which does not play nice with tqdm. thanks for the tip about positions though, that helped me search more effectively and came up with two promising comments. unmerged / require some workarounds, but might just work:

https://github.com/tqdm/tqdm/issues/1000#issuecomment-184208... https://github.com/tqdm/tqdm/issues/811#issuecomment-1368850...

clever! will have to see if this works with tqdm progress bars, has anyone tried that?

> 30b+ parameter model doing RAG as part of a conversation with voice responses in less than a second, running on Nvidia.

I believe that this is doable - my pipeline is generally closer to 400ms without RAG and with Mixtral, with a lot of non-ML hacks to get there. It would also definitely be doable with a joint speech-language model that removes the transcription step.

For these use cases, time to first byte is the most important metric, not total throughput.

whisper is simply not designed for this, in many ways, and it's impressive engineering to try and overcome its limitations, but I can't help but feel that it is easier to just use an architecture that is designed for the problem.

I was impressed by Kaldi's models for streaming ASR: https://k2-fsa.github.io/sherpa/onnx/pretrained_models/index... ; I suspect that the Nvidia/Suno Parakeet models will also be pretty good for streaming https://huggingface.co/nvidia/parakeet-ctc-0.6b

their own shortener, e.g. fb.me, presumably

yes (you skip a decoding step) but also no (when do you start emitting?)

> See Midjourney's reddit activity after DALLE-3

What stats are you looking at? Looking at https://subredditstats.com/r/midjourney , I see a slower growth curve after the end of July, but still growing and seemingly unrelated to the DALL-E 3 release, which was more like end of October publicly.

do you construct an index over the superbit signatures to perform approximate search or do you perform exact search?

cheaper prices -> more usage -> larger batch sizes / better gpu utilization -> lower cost of service

curious what the latency impact is to apply the dimensionality reduction... may hint at the specific technique they use

I don't think it does. And there is a pretty big risk that you end up picking up on some quirk ("bias") of your reward model that doesn't reflect reality -- GPT4 preferring longer answers is one such commonly observed bias. AFAIK there is not a great theoretical basis for why we can avoid mode collapse, except empirically the models are good enough to survive some bootstrapping.

I did a similar optimization via https://github.com/viterin/vek as the SIMD version. Some somewhat unscientific calculations showed a 10x improvement staying in float32: https://github.com/stillmatic/gollum/blob/07a9aa35d2517af8cf... (comparable to the 9x improvement in article using SIMD + int8)

TBH my takeaway was that it was more useful to use smaller vectors as a representation

My interpretation is that they prohibit using distilled models to compete against OpenAI, i.e. to offer a foundation model as a service. This particular app is a product not a foundation model so seems pretty clearly fine to me.

Of course, this restriction is commonly ignored (eg 'open source' inference server companies offering distilled models) and who knows what applied products OpenAI will eventually build.

you could make an informed guess by looking at what the highest quality open source model is, looking at the current employer of that model's creator, and what they currently work on there

bfloat16 mostly matters for training stability, not for inference. M1 is inefficient for training any reasonably sized model and most inference efforts target 4-8 bits, even.

the number 1 in Mandarin Chinese is usually pronounced "yi" which can be mistaken for the number 7 ("qi"). so in certain contexts, e.g. reading out a phone number, it would be pronounced "yao" instead.

agree, this is a nice example of generating synthetic data, and I believe that the synthetic data is helpful for generating useful embeddings for RAG, but not including an ablation with fine-tuned E5 or another commonly used embedding model (to control for the 'bigger model wins' effect) is a glaring omission. this paper shares many authors with the E5 paper, why did they not compare on a fair basis?

the BTL model is just a way to infer 'true' skill levels given some list of head to head comparisons. the head to head comparisons / rankings are the most important!!!! and in this case, the rankings come from GPT-4 itself. so take any subsequent score with all the grains of salt you can muster.

their methodology also appears to be 'try 12 different models and hope 1 of them wins out.' multiple hypothesis adjustments come to mind here :)

some catches I'm looking for: it remains to be seen if the generations are high quality (not just low perplexity) and if the architecture scales beyond 3B, but definitely super promising