Insanely fast whisper is mainly taking advantage of a GPU’s parallelization capabilities by increasing the batch size from 1 to N. I doubt it would meaningfully improve CPU performance unless you’re finding that running whisper sequentially is leaving a lot of your CPU cores idle/underutilized. It may be more complicated if you have a matrix co-processor available, I’m really not sure.
Our comparisons were a little while ago so I apologize I can’t remember if we used BS 1 or 5 - whichever we picked, we were consistent across models.
Insanely fast whisper (god I hate the name) is really a CLI around Transformers’ whisper pipeline, so you can just use that and use any of the settings Transformers exposes, which includes beam size.
We also deal with very poor audio, which is one of the reasons we went with faster whisper. However, we have identified failure modes in faster whisper that are only present because of the conditioning on the previous segment, so everything is really a trade off.
To be fair, defense is an existential risk for the US and its allies. NATO can’t really afford to not have a reasonably up-to-date combat jet. They also need to continually feed money into the military industrial complex so that suppliers don’t go under/downsize too much/etc.
Not disagreeing with your sentiment, just think that certain fields like defense, healthcare, etc have slightly different priority lists.
I think this is using the OpenAI Whisper repo? If they want a real comparison, they should be comparing MLX to faster-whisper or insanely-fast-whisper on the 4090. Faster whisper runs sequentially, insanely fast whisper batches the audio in 30 second intervals.
We use whisper in production and this is our findings: We use faster whisper because we find the quality is better when you include the previous segment text. Just for comparison, we find that faster whisper is generally 4-5x faster than OpenAI/whisper, and insanely-fast-whisper can be another 3-4x faster than faster whisper.
As another commenter pointed out, you can give context for the decoder. So you can feed previous chunks into the model as the context. This is how we do it for streaming, at least.
I think the Ars writer here is very uninformed. Floc is completely different from Topics, and if you actually read the Topics spec, it seems to be significantly better than 3rd party cookies? At least to me. Maybe I’m missing something.
That’s a good question and I can’t give a straight answer, because we are actually on three year lease plans through a vendor (think HPE). I know we get a pretty decent deal in general from them since we have basically an entire data center worth of hardware from them, but I don’t know how good the deal is on the GPUs, since they’re in such high demand.
I think the author has missed a pretty large segment of demand. Non-cloud/non-tech enterprises are also buying large quantities of H100s and A100s for their own machine learning and simulation workloads. Where I work, we are going to have more than 1000 H100s by the end of the year, I am very excited to start benchmarking them soon :)
It does seem much faster and significantly worse to me - but I haven’t set any repeatable benchmark queries for myself, I suppose I could be imagining it. I hope they bring back the slow version. I don’t care if they have to cut the limit down to 20 messages every 3 hours, using that slow version of GPT4 it felt like I had a very competent coworker. Now it’s feeling more like a sometimes useful chatbot. Real shame.
Agreed. I am so trained by macOS that it needs my iCloud password or account password randomly that I don’t even think about it anymore. I like to consider myself a savvy computer user, but I frankly only give myself a 50% chance at best of identifying a malicious password prompt at this point.
I think that’s missing one big piece. What people really want cheap education that ends with a degree that is as well regarded as an Ivy or good state college degree. Community colleges, for better or worse, are seen as inferior education. So they are not equivalent to simply a cheaper option of more expensive schools.
I agree that I’m uncomfortable with the consolidation of so much of the HPC stack under Nvidia (and would be 10x more uncomfortable if the ARM acquisition went through), but I don’t think there was really a monopoly there. I think the real “market” is networking, not just infiniband, and in that sense there is a ton of competition, just no competition that hits the performance characteristics that Nvidia/Melanox achieve.
You’re correct, I was going to add on to my answer that this is combined with DDP for a “3D” style parallelism that could specifically benefit from the TPU’s network topology, but by the time I got done fixing all the typos (and still missing a few) from writing it on my phone I completely forgot :)
Just as a quick example (there are lots of other examples), I currently am working on a model that does significant tensor-level parallelism. The input to the model is huge, so we replicate the weights for the encoder part of the model over 12 A100s, and then run an all-reduce over the outputs before then distributing that reduced tensor to another set of GPUs for the next set of operations. In this setup we need significant bandwidth not only between every device in a node but also across nodes (infiniband).
Apologies for being off topic but does anyone know technically how they’ve implemented their 3 modes (creative balanced and… was it factual?). Is it just adjusting the prompt? Or are they doing something like adjusting the temperature, or fine-tuning the output layers?
Beyond the other answers, I’ll point out that pytorch is developing tools that will make doing this work by hand or implementing in a framework much easier. They’re building a native DTensor implementation and testing out SPMD-style distributed models with pipelining. DTensor is in torch.distributed, and the SPMD code is in the repo called Tau under the pytorch org on github.
Similar setup here. I use a Mac laptop for all my general purpose work/programming/leisure, Linux servers I ssh into from the Mac for running jobs (ML stuff), and a windows laptop for playing games. Obviously a luxury to be able to have all three, but I think it allows me to use all three for the things they’re best at.
I think I’d switch that from “quickly” to “eventually”, or “have a head start to” - we could get grid independence “relatively” quickly if the government subsidized it (I highly doubt first Gen fusion competes with natural gas or solar cost-wise), but a large amount of energy is used in transportation, home heating, etc.. Until those become fully electrified you’re still stuck in the fossil fuel economy.
I don’t see how Twitter could afford to compete with YouTube financially. YouTube has the larger audience (3x MAU) so they get the best advertising deals, and they STILL don’t break out YouTube’s profit/loss individually, suggesting it’s probably either low margin or a loss leader for alphabet.
I agree with the spirit of your comment, and I am extremely excited by these results. However, I think the history of fusion has showed us that the cynics have had a much better track record than the fusion optimists, haha.
My very uninformed opinion (nuclear physicist by training, but not specialized in fusion, lasers, or plasma physics) is that we’re still 20 years (haha) away from fusion energy making its way into the power grid. And that is assuming this result (or other things, like the relative instability of global energy markets lately) causes an increase in funding for the field so that they can solve all the pesky engineering issues related to efficiency, reactor lifespan, reliability, cycling speed, etc.
That’s interesting. I have a MacBook for work with no Apple ID and I don’t think I’ve ever been asked to sign into anything. I wonder if there are fleet management controls my company is using to disable a lot of that stuff.
So if I’m parsing the results correctly they only have a single data point per language? Seems like they are testing individual programmers styles/speed just as much as they are testing the languages.