I have 20 years of software development experience, and I couldn’t understand anything you said. Is there a dictionary for this new lingo, or am I just too mid?
Also mixtral medium - no idea of what he means by that.
Not to mention a claim that mixtral is as good as gpt-4. It’s on the quality of gpt3.5 at best, which is still amazing for an open source model, but a year behind openai
Edit: I was using 2k window, a larger one would probably eat more ram. But even with 2k it didn’t feel like it loses context or something.
As these models can be quite large and memory intensive, if you want to just give it a quick spin, huggingface.co/chat, chat.nbox.ai, and labs.pplx.ai all have Mixtral hosted atm.
"mixtral medium" is just a typo: he means mistral-medium.
And GPT 4.5 is certainly not an "invention of his". Whether it exists or not (which is debatable, OpenAI said it was just mentioned in a GPT 4 hallutination and caught on), it' s a version name thrown around for like a month in forums, blog posts, news articles and such.
I claimed it's not an "invention of his [benreesman ]", but a term that was already out there.
For a broad introduction to the field Karpathy's YouTube series is about as good as it gets.
If you've got a pretty solid grasp of attention architectures and want a lively overview of stuff that's gone from secret to a huge deal recently I like this treatment as a light but pretty detailed podcast-type format: https://arize.com/blog/mistral-ai
.. which seemed to fit suprisingly well.
- I know r/LocalLlama, huggingface's Daily Papers and TheBloke. Most of what Youtube throws at me is horrific clickbait. I feel like there are probably whole communities I'm missing out on.
lulz.
Karpathy has a a great YouTube series where he gets into the details from `numpy` on up, and George Hotz is live-coding the obliteration of PyTorch as the performance champion on the more implementation side as we speak.
Altman being kind of a dubious-seeming guy who pretty clearly doesn't regard the word "charity" the same way the dictionary does is more-or-less common knowledge, though not often mentioned by aspiring YC applicants for obvious reasons.
Mistral is a French AI company founded by former big hitters at e.g. DeepMind that brought the best of the best on 2023's public domain developments into one model in particular that shattered all expectations of both what was realistic with open-weights and what was possible without a Bond Villain posture. That model is "Mixtral", an 8-way mixture of experts model using a whole bag of tricks but key among them are:
- gated mixture of experts in attention models - sliding window attention / context - direct-preference optimization (probably the big one and probably the one OpenAI is struggling to keep up with, probably more institutionally than technically as probably a bunch of bigshots have a lot of skin in the InstructGPT/RLHF/PPO game)
It's common knowledge that GPT-4 and derivatives were mixture models but no one had done it blindingly well in an open way until recently.
SaaS companies doing "AI as a service" have a big wall in front of them called "60%+ of the TAM can't upload their data to random-ass cloud providers much less one run by a guy recently fired by his own board of directors", and for big chunks of finance (SOX, PCI, bunch of stuff), medical (HIPAA, others), defense (clearance, others), insurance, you get the idea: on-premise is the play for "AI stuff".
A scrappy group of hackers too numerous to enumerate but exemplified by `ggerganov` and collaborators, `TheBloke` and his backers, George Hotz and other TinyGrad contributors, and best exemplified in the "enough money to fuck with foundation models" sense by Mistral at the moment are pulling a Torvalds and making all of this free-as-in-I-can-download-and-run-it, and this gets very little airtime all things considered because roughly no one sees a low-effort path to monetizing it in the capital-E enterprise: that involves serious work and very low shady factors, which seems an awful lot like hard work to your bog-standard SaaS hustler and offers almost no mega data-mining opportunity to the somnobulent FAANG crowd. So it's kind of a fringe thing in spite of being clearly the future.
It currently ranks 4 in chatbot arena leaderboard (slightly behind GTA-4 ELO rating): https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboar...
Hilariously neither knows who is sama (Sam Altman, the Drama King of OpenAI), nor do they recognize when they themselves are being discussed.
Reading the responses in full also gives you a glimpse on specific merits or weaknesses of these systems, namely how up to date is their knowledge and lingo, explaining capabilities, and ability to see through multiple layers of referencing. Also showcases whether the AIs are willing to venture guessing to piece together some possible interpretation for hoomans to think about.
I screen-capped my take on this to prove* that I was actually wiring all this stuff up and plug my nascent passion/oss project, but it's really funny comparing them either way: https://imgur.com/WDrqxsz
Basically, he said he is happy with Mistral 8x7B and thinks it is on par/better comparing to OpenAI's closed source model.
You have set up your system to run different AI models and compare their performance using a text editor. You are using Mixtral-8x7, a high-quality open-source model developed by Mistral AI, Dolphin, an emulator for Nintendo video games, 3.5-Turbo, a customized version of GPT-3.5, a powerful natural language model, and 4-Series Preview, a new version of the BMW sports coupe. You have noticed that the 4.5-Preview, an upcoming update of GPT-3.5, is slightly better than Mixtral-8x7, which used to be a close match. You are still waiting to access Mistral-Medium, a prototype model that is even better than Mixtral-8x7, but only available to a limited number of users.
You have discovered that Perplexity, an AI company that provides information discovery and sharing services, offers free access to Mistral-Medium through their partnership with Mistral AI. You think that Perplexity is making a mistake by giving away such a valuable model, and that they are underestimating the superiority of Mistral-Medium over the 4.5-Preview. You also think that Mistral AI is the new leader in the AI industry, and that their techniques, such as DPO (Data Processing Optimization), Alibi (a library for algorithmic accountability), sliding window (a method for analyzing time series data), and modern mixtures (a way of combining different models), are well-known and effective. You believe that the advantage of Mistral AI lies in the gap between their innovation and the ability of other developers to replicate it on cheaper and more accessible hardware. You also think that the enterprise market is not fond of the complex structure of GPT-3.5 and its variants, and that they prefer to use Mistral AI's models, which are more affordable and operable on their own premises.
You end your text with a quote from the movie Armageddon, which implies that you are leaving a situation that you dislike, but also admire.
What I am confused about though is it seems like the parent is mentioning models beyond the GPT4 instance I currently have access to. I checked their twitter and I have seen no anouncement for any 4.5 or 4 series previews. Is this just available to people using the API or did I miss something?
I don't recall if I saw any press calling anything `4.5`, but it's a different model in some important ways (one suspects better/cheaper quantization at a minimum) and since they've used `.5` for point releases in the past it seemed the most consistent with their historical versioning.
Well, the Paul Ricard circuit in France has a straight called Mistral. Plenty of BMWs have been there for sure, and a zillion other cars.
I wonder if that could have confused the AI a little in combination with other hints. Turbo?
If that's a thing maybe we should start picking our names not only to make them googlable but also not to confuse LLMs at least for the next few years. Months?
It's heavily integrated with my custom model server and stuff and I'm slowly getting it integrated with other leading tools (vscode and nvim and stuff).
I plan to MIT it all once it's at a reasonable RC. If I get there it will be available at `https://hyper-modern.ai` and `https://github.com/hyper-modern-ai`.
Thanks for asking!
Give me a hand?
am just here to vibe in my down time