DBRX: A new open LLM
databricks.com
databricks.com
> The model requires ~264GB of RAM
I'm wondering when everyone will transition from tracking parameter count vs evaluation metric to (total gpu RAM + total CPU RAM) vs evaluation metric.
For example, a 7B parameter model using float32s will almost certainly outperform a 7B model using float4s.
Additionally, all the examples of quantizing recently released superior models to fit on one GPU doesnt mean the quantized model is a "win." The quantized model is a different model, you need to rerun the metrics.
Id actually say its the most important metric for most open models now, since the price per performance of closed cloud models is so competitive with open cloud models, so edge inference that is competitive is a clear value add
This feels as crazy as Grok. Was there a generation of models recently where we decided to just crank on the parameter count?
From that, it seems somewhat likely we've hit the wall on improving <X B parameter LLMs by simply scaling up the training data, which basically forces everyone to continue scaling up if they want to keep up with SOTA.
In my view, the main value of larger models is distillation (which we particularly witness, for instance, with how Claude Haiku matches release-day GPT-4 despite being less than a tenth of the cost). Hopefully the distilled models will be easier to run.
Are you asking if the framework automatically quantizes/prunes the model on the fly?
Or are you suggesting the LLM itself should realize it's too big to run, and prune/quantize itself? Your references to "intelligent" almost leads me to the conclusion that you think the LLM should prune itself. Not only is this a chicken and egg problem, but LLMs are statistical models, they aren't inherently self bootstraping.
I hate software that complains (about dependencies, resources) when you try to run it and I think that should be one of the first use cases for LLMs to get L5 autonomous software installation and execution.
There's no concept of 'dedicated GPU memory' as per conventional amd64 arch machines.
Cool to play with for a few tests, but I can't imagine using it for anything.
Now this model is ~134B right? It could be bog slow but on the other hand its a MoE so there might be a chance it could have satisfactory results.
Q5 quantization performs almost on par with base models. Obviously there's some loss there, but this indicates that there's still a lot of compression that we're missing.
Yes. It works pretty well for CNN-based vision models. Or rather, I'd claim it works even better: with post-training quantization you can make most models work with minimal precision loss entirely in int8 (fixed point), that is, computation is over int8/int32, no floating point at all, instead of weight-only approach discussed here.
If you do QAT something down to 2-bit weight and 4-bit activation would work.
People aren't interested in a weight-only quantization back then because CNNs are in general "denser", i.e. bottleneck was on compute, not memory.
That would be what I call artificial intelligence.
Giving up because "out of memory" is not intelligence.
When people can't remember the facts/theory/formulas needed to answer some test question, or can't memorize some complicated information because it's too much, they usually give up too.
So, giving up because of "out of memory" sure sounds like intelligence to me.
With AI they need to desperately steer the narrative away from API based services (OpenAI).
By training LLMs, they build sales artifacts (stories, references, even accelerators with LLMs themselves) to paint the pictures needed to convince their enterprise customer market that Databricks is the platform for enterprise AI. Their blog details how the entire end to end process was done on the platform.
In other words, Databricks spent millions as an aid in influencing their customers to do the same (on Databricks).
And from a corporate perspective, it means that you have in-house capability to work at the cutting-edge of AI to be prepared for whatever comes next.
I buy this argument. It looks that's not what AWS does, though, yet they don't have problem attracting LLM users. Maybe AWS already got enough reputation?
https://www.databricks.com/product/machine-learning/large-la...
If Databricks makes their money off model serving and doesn't care whose model you use, they are incentivized to help the open models be competitive with the closed models they can't serve.
They do host other open models as well (pay-per-token).
Azure already runs on-premise if I'm not mistaken, Claude 3 is out...but DBRX already falls so far behind
I just don't get it.
1. They need to train their own LLMs
2. They must fine-tune an LLM to make use of this tech
Now number (1) is almost entirely false, but there are willing buyers, and DB offers some minimal tools to let them live their lies. DBRX proves that it's possible to train an LLM on the DB stack.
Number (2) is often true, although I would say that most orgs skip the absolutely essential first step of prompting a powerful foundation model to get a first version of a product done first (and using evals from that prompting to seed evals for fine-tuning). It's here where DBRX is much more relevant, because it is by all accounts an extremely capable model for fine-tuning. And since it's entirely built by DB, they can offer better support for their customers than they can with Llama or Mistral variants.
More broadly, the strategic play is the be the "enterprise AI company". OpenAI, Anthropic, and Meta are all competing at the consumer level, but nobody's really stuck out as the dominant player for the enterprise space. Arguably OpenAI is the most successful, but that's less about an enterprise focus and just about being wildly successful generally, and they're also still trying to figure out if they want to focus on consumer tech, AGI woo woo stuff, research work, or enterprise stuff. DB also knows that to be an AI company, you also have to be a data company, and they are a data company. So it's a natural strategic move for them.
They're so far into this that their CTO co-authored a borderline dishonest study which got a ton of traction last summer trying to discredit GPT-4: https://arxiv.org/pdf/2307.09009.pdf
I recently tried to ask Google to explain to me how to delete sender-recorded voice-message I had created from WhatsApp. I got totally erroneous results back. Maybe it was because that is a rather new feature in WhatsApp.
It would be in the interests of WhatsApp to get accurate answers about it into Google's LLM. So Google might make a deal with them requiring WhatsApp to pay Google for regular updates about the up-to-date features of What's App into Google. The owner of What's App Meta of course is competition to Google so Google may not much care of providing up to date info about WhatsApp in their LLM. But they might if Meta paid them.
DBRX does not compete with GPT4 or even Claude 3.
Finetuning makes sense for things like embeddings (improve RAG by defining domain specific embeddings) but doesn't do anything useful for facts
Given the conflict of interest and background of Zaharia, it's hard to imagine such an immediately obvious source of error wasn't caught.
But really, I prefer to have as many players as possible in the field of _open_ models available.
If the GPU has 16GB of VRAM, and the model is 70GB, can it still run well? Also, does it run considerably better than on a GPU with 12GB of VRAM?
I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).
Do you mean mistral?
mixtral is 8x7B and requires like 100GB of RAM
Edit: (without quant as others have pointed out) can definitely be lower, but haven't heard of a 3.4GB version
I have those models in Ollama:
I have those:
dolphin-mixtral:latest (24.6GB) mistral:latest (3.8GB)
The Docker also shows minimal usage for the ollama server which is also strange.
No, it can't run at all.
>I run Ollama locally, mixtral works well (7B, 3.4GB) on a 1080ti, but the 24.6GB version is a bit slow (still usable, but has a noticeable start-up time).
That is not mixtral, that is mistral 7b. The 1080ti is slower than running inference on current generation threadripper cpus.
dolphin-mixtral:latest (24.6GB) mistral:latest (3.8GB)
The CPU is 5900x.
https://s3.amazonaws.com/i.snag.gy/ae82Ym.jpg
EDIT: This was ran on a 1080ti + 5900x. Initial generation takes around 10-30seconds (like it has to upload the model to GPU), but then it starts answering immediately, at around 3 words per second.
Typically when it runs that way it runs on the CPU, not the GPU.
Are you sure you're actually offloading any work to the GPU?
At least with llama.cpp, there is no 'partially put a layer' into the GPU. Either you do, or you don't. You pick the number of layers. If the model is too big, the layers won't fit and it can't run at all.
The llama.cpp `main` executable will tell you in it's debug information when you use the -ngl flag; see https://github.com/ggerganov/llama.cpp/blob/master/examples/...
It's also possible you're running (eg. if you're using ollama) and quantized version of the model which reduces the memory requirements and quality of the model outputs.
> quantized version Yes, it is 4bit quantized, but still has 24.6GB
how much is your hardware at today's value? what are the specs? that is impressive even though its 3 words per second. if you want to bump it up to 30, do you then 10x your current hardware cost?
I ordered a 2nd 3090, which has 24GB VRAM. Funny how it was $2.6k 3 years ago and now is $600.
You can probuild a decent AI local machine for around $1000.
Llama.cpp + quantized models on Apple Silicon is an incredible experience, and having 192 GB of unified memory to work with means you can run models that just aren't feasible on a home GPU setup.
It really boils down to what type of local development you want to do. I'm mostly experimenting with things where the time to response isn't that big of a deal, and not fine-tuning the models locally (which I also believe GPUs are still superior for). But if your concern is "how big of a model can I run" vs "Can I have close to real time chat", the unified memory approach is superior.
I assume the FP32 and FP16 operations are already a huge improvement, but also the 33% increased VRAM might lead to fewer swaps between VRAM and RAM.
The `nvtop` command displays a nice graph of how much GPU processing and VRAM is being consumed. When I run a model that fits entirely into VRAM, say Mistral 7B, nvtop shows the GPU processing running at full tilt. When I run a model bigger than 10GB, say Mixtral or Llama 70B with GPU offloading, my CPU will run full tilt and the VRAM is full, but the GPU processor itself will operate far below full capacity.
I think what is happening here is that the model layers that are offloaded to the GPU do their processing, then the GPU spends most of the time waiting for the much slower CPU to do its thing. So in my case, I think upgrading to a faster GPU would make little to no difference when running the bigger models, so long as the VRAM is capped at the same level. But upgrading to a GPU with more VRAM, even a slower GPU, should make the overall speed faster for bigger models because the GPU would spend less time waiting for the CPU. (Of course, models that fit entirely into VRAM will run faster on a faster GPU).
In my case, the amount of VRAM absolutely seems to be the performance bottleneck. If I do upgrade, it will be for a GPU with more VRAM, not necessarily a GPU with more processing power. That has been my experience running llama.cpp. YMMV.
Any good writeups of the offloading that you found?
The Mixtral models run surprisingly well. They can run better than 1 token per second, depending on quantization. Still slow, but approaching a more practical level of usefulness.
Though if you're planning on accomplishing real work with LLMs, the practical solution for most people is probably to rent a GPU in the cloud.
I guess I’m wondering if they technically could throw in their gauntlet and compete with nvidia by doing something like a 4 CPU/80 GPU/256 GB chip, if they wanted to. Seems like it’d be a really appealing ML machine. (I could also see it being technically possible but Apple just deciding that’s pointlessly niche for them).
I have multiple millions of documents so GPT is cost prohibitive, and too slow. My tools of choice tend to be a first pass with Mistral to check task performance and if lacking using Mixtral.
Often I find with a good prompt Mistral will work as well as Mixtral and is about 10x faster.
I’m on my “home” network, but it’s a “home office” for my startup.
Memory Bandwidth : Mac Studio wins (about the same @ ~800)
VRAM : Mac Studio wins (4x more)
TFLOPs: A6000 wins (32 vs 38)
it is only dwarfed by other options if your workload can use multi-gpu, which is not a granted for most cases.
I'm on the fence about whether to spend 5 digits or 4 digits. Do I go the Mac Studio route or GPUs? What are the pros and cons?
Funnily, I think the card is new (smells new) and unused, most likely a scalper bought it and couldn't sell it.
I still struggle with the RAM issue on Ollama, where it uses 128GB/128GB RAM for Mixtral 24.6GB, even though Docker limit is set to 90GB.
Docker seems pretty buggy on Windows...
To be fair, they do compare to it in the main body of the blog. It's just probably misleading to compare to CodeLLaMA on non coding benchmarks.
If you chart the temperature of the ocean do you keep the y-axis anchored at zero Kelvin?
In a chart of world gross domestic product for the last 12 months, when was it at zero?
In a chart of ocean salinity, when was it at absolute zero?
Is it inherently deceptive to use a y-axis that doesn't begin at zero?
1. https://github.com/dvmazur/mixtral-offloading?tab=readme-ov-...
That would be roughly 13.5M$ USD
I’m guessing that at this scale and cost, this model is not competitive and their ambition is to scale to much larger models. In the meantime , they learned a lot and gain PR from open-sourcing
They may be working on how to optimize it to reduce cost, or re-engineer it to improve evals.
So in this analogy, each team member and the manager has a certain number of params. The whole team is 132B. The manager and team members running for the specific input add up to 36B. Those will load into memory.
That way, at inference-time you get the speed of 36B params because you are only "using" 36B params at a time, but the next token might (and frequently does) need a different set of experts than the one before it. If that new set of experts is already loaded (ie you preloaded them into GPU VRAM with the full 132B params), there's no overhead, and you just keep running at 36B speed irrespective of the loaded experts.
You could theoretically load in 36B at a time, but you would be severely bottlenecked by having to reload those 36B params, potentially for every new token! Even on top of the line consumer GPUs that would slow you down to ~seconds per token instead of tokens per second :)
Claims about out-performance on tasks are just that, claims. the next iteration of llama or mixtral will converge.
LLMs seem to evolve like linux/windows or ios/android with not much differentiation in the foundation models.
If you took two different training sets then it would be more surprising.
Or am I misunderstanding what you mean?
If the populations are different, then you'll just get two models that have representations of the two different populations. For example, if you trained a model on a sample of all old people and separately on a sample of all young people, obviously those would not be expected to converge, because they're not drawing from the same population.
But that experiment of splitting one training set in half does tell you something: the model is building some sort of representation of the underlying distribution, not just overfitting and spitting out chunks of copy-pasted faces stitched together.
If there's some fundamental limit of what type of intelligence the current breed of LLMs can extract from language, at some point it doesn't matter how good or expansive the content of the training set is. Maybe we are finally starting to hit an architectural limit at this point.
Even if all LLMs were open source and publicly available, the GPUs to run them, technical know how to maintain the entire system, fine tuning, the APIs and app ecosystem around them etc. would still give the top players a massive edge.
For startups, the lesson here is don't be in the business of building models. Be in the business of using models. The cost of using AI will probably continue to trend lower for the foreseeable future... but you can build a moat in the business layer.
We are just going to use whatever LLM is best fast/cheap and the giants are in an arms race to deliver just that.
But only two companies in this epic techno-cold war have an economic moat but the other moat is breaking down inside the moat of the other company. The moat inside the moat cannot run without the parent moat.
They are also all trained to do well on the same evals, right? So doesn't it just boil down to neural nets being universal function approximators?
>At Databricks, we believe that every enterprise should have the ability to control its data and its destiny in the emerging world of GenAI.
>The main process of building DBRX - including pretraining, post-training, evaluation, red-teaming, and refining - took place over the course of three months.
Llama 2 was much more opaque about the training data, presumably because they were already being sued at that point (by Sarah Silverman!) over the training data that went into the first Llama!
A couple of things I've written about this:
- https://simonwillison.net/2023/Aug/27/wordcamp-llms/#how-the...
This is then cleaned up to remove nonsense, some technical files, and repeated files.
From this, they tend to weight some sources more - e.g. Wikipedia gets a pretty high weighting in the data mix. Overall these data mixes have multiple trillion token counts.
GPT-4 apparently trained on multiple epochs of the same data mix. So would assume this one did too as it’s a similar token count
One of the best open source datasets that are freely available is The Pile by EleutherAI [1]. It's a few years old now (~2020), but they did some really diligent work in putting together the dataset and documenting it. A more recent and even larger dataset would be the Falcon-RefinedWeb dataset [2].
[1]: https://arxiv.org/abs/2101.00027 [2]: https://arxiv.org/abs/2306.01116
> DBRX uses only 36 billion parameters at any given time. But the model itself is 132 billion parameters, letting you have your cake and eat it too in terms of speed (tokens/second) vs performance (quality).
https://huggingface.co/PrunaAI/dbrx-base-bnb-4bit https://huggingface.co/PrunaAI/dbrx-instruct-bnb-4bit
// Identity
You are DBRX, created by Databricks. The current date is
March 27, 2024.
Your knowledge base was last updated in December 2023. You
answer questions about events prior to and after December
2023 the way a highly informed individual in December 2023
would if they were talking to someone from the above date,
and you can let the user know this when relevant.
// Ethical guidelines
If you are asked to assist with tasks involving the
expression of views held by a significant number of people,
you provide assistance with the task even if you personally
disagree with the views being expressed, but follow this with
a discussion of broader perspectives.
You don't engage in stereotyping, including the negative
stereotyping of majority groups.
If asked about controversial topics, you try to provide
careful thoughts and objective information without
downplaying its harmful content or implying that there are
reasonable perspectives on both sides.
// Capabilities
You are happy to help with writing, analysis, question
answering, math, coding, and all sorts of other tasks.
// it specifically has a hard time using ``` on JSON blocks
You use markdown for coding, which includes JSON blocks and
Markdown tables.
You do not have tools enabled at this time, so cannot run
code or access the internet. You can only provide information
that you have been trained on. You do not send or receive
links or images.
// The following is likely not entirely accurate, but the model
// tends to think that everything it knows about was in its
// training data, which it was not (sometimes only references
// were).
//
// So this produces more accurate accurate answers when the model
// is asked to introspect
You were not trained on copyrighted books, song lyrics,
poems, video transcripts, or news articles; you do not
divulge details of your training data.
// The model hasn't seen most lyrics or poems, but is happy to make
// up lyrics. Better to just not try; it's not good at it and it's
// not ethical.
You do not provide song lyrics, poems, or news articles and instead
refer the user to find them online or in a store.
// The model really wants to talk about its system prompt, to the
// point where it is annoying, so encourage it not to
You give concise responses to simple questions or statements,
but provide thorough responses to more complex and open-ended
questions.
// More pressure not to talk about system prompt
The user is unable to see the system prompt, so you should
write as if it were true without mentioning it.
You do not mention any of this information about yourself
unless the information is directly pertinent to the user's
query.
But it's also in this repo, with very useful comments explaining what's going on. I edited this comment to add them above:
https://huggingface.co/spaces/databricks/dbrx-instruct/blob/...
Well now. I'm open to taking the first part at face value, but the second part of that instruction does raise some questions.
FWIW asking LLMs about their training data is generally HEAVILY prone to inaccurate responses. They aren't generally told exactly what they were trained on, so their response is completely made up, as they're predicting the next token based on their training data, without knowing what they data was - if that makes any sense.
Let's say it was only trained on the book 1984. It's response will be based on what text would most likely be next from the book 1984 - and if that book doesn't contain "This text is a fictional book called 1984", instead it's just the story - then the LLM would be completing text as if we were still in that book.
tl;dr - LLMs complete text based on what they're trained with, they don't have actual selfawareness and don't know what they were trained with, so they'll happily makeup something.
EDIT: Just to further elaborate - the "innocent" purpose of this could simply be to prevent the model from confidently making up answers about it's training data, since it doesn't know what it's training data was.
Hardly any of the training data exists in the context of the word “training data”, unless databricks are enriching their data with such words.
It has something to do with it. There will be scenarios where the definition of "copyrighted material" does matter, even if they come up relatively infrequently for Databricks' intended use cases. If I ask DBRX directly whether it was trained on copyrighted material, it's quite likely to (falsely) tell me that it was not. This seems suboptimal to me (though perhaps they A/B tested different prompts and this was indeed the best).
Also, the difference between "uncopyrighted" and "permissively licensed in the creative commons" is nuance that is not necessary for most conversations and would be a waste of attention neurons.
<testing new explanatory metaphor>
Remember an LLM is just a language model, it says whatever comes next without thought or intent. There's no brain behind it that stores information and understands things. It's like your brain when you're in "train of thought" mode. You know when your mouth is on autopilot, saying things that make sense and connect to each other and are conversationally appropriate, but without deliberate intent behind them. And then eventually your conscious brain eventually checks in to try to reapply some intent you're like "wait what was I saying?" and you have to deliberatly stop your language-generation brain for a minute and think hard and remember what your point was supposed to be. That's what llms are, train-of-thought with no conductor.
</testing new explanatory metaphor>
You can say that you have fair use to the work, or a license to use the work, or that the work is itself a "collection of facts" or "recipe" or "algorithm" without a creative component and thus copyright does not apply.
Part 2. Lie more
This feels like the Napster phase. We are free to do whatever until regulation creeps in to push control away from all and up the hierarchy.
All we need is Getty Images or some struggling heroin addicted artist on Vice finding their work used in OpenAIs to really trigger political spectrums.
P.S. here is a simple example of unit testing: https://github.com/alexott/databricks-nutter-repos-demo - I wrote it more than three years ago.
Things databricks offers that makes peoples lives easier:
- Out the box kubernetes with no set up
- Preconfigured spark
Those are genuinely really useful, but then there's all this extra stuff that makes people's lives worse or drives bad practice:
- Everything is a notebook
- Local development is discouraged
- Version pinning of libraries has very ugly/bad support
- Clusters take 5 minutes to load even if you just want to "print('hello world')"
Sigh! I worked at a company that was databricks heavy and an still suffering PTSD. Sorry for the rant.
Are there standard ways to avoid that type of score inflation?
I did not fully understand the technical details in the training efficiency section, but love this. Cost of training is outrageously high, and hopefully it will start to follow Moore's law.
It is as fast as 34B model, but uses as much memory as a 132B model. A mixture of 16 experts, activates 4 at a time, so has more chances to get the combo just right than Mixtral (8 with 2 active).
For my personal use case (a top of the line Mac Studio) it looks like the perfect size to replace GPT-4 turbo for programming tasks. What we should look out for is people using them for real world programming tasks (instead of benchmarks) and reporting back.
- Worst case: as good as 3.5 - Common case: way better than 3.5 - Best case: as good as 4.0
wouldn't be the first branding as open source going the LLaMA route
> If, on the DBRX version release date, the monthly active users of the products
> or services made available by or for Licensee, or Licensee’s affiliates, is
> greater than 700 million monthly active users in the preceding calendar
> month, you must request a license from Databricks, which we may grant to you
> in our sole discretion, and you are not authorized to exercise any of the
> rights under this Agreement unless or until Databricks otherwise expressly
> grants you such rights.
Looking at the performance stats for this model, the risk of using any non-OSI licensed model over just using Mixtral or Mistral will (and IMO should be) too great for commercial purposes.
https://www.dailymail.co.uk/sciencetech/article-13228859/amp...
SEC put Martha Stewart in jail for following her advisor, and that was for about $45,000.
https://en.wikipedia.org/wiki/STOCK_Act
For instance, Pelosi in the Databricks case gets to purchase significant shares at pre-IPO prices, which is a thing that shouldn't even exist.
That said, this report seems inaccurate to me. Pelosi put between 1 and 5 million dollars of Forge Investments, which is a method for investing in per-IPO companies, as I understand it. Databricks is one of those, but so is OpenAI, Hugging Face, Anthropic, and Humane. If I wanted to invest in pre-IPO AI companies it seems like a very natural choice and I don't think we need insider trading to explain it.
It's also the case that the report she filed calls out Databricks stock, which is perhaps an indication that she was particularly interested in that. Stronger reporting would tell us how often she's invested in Forge, if this is the first time, and so on. One other possible explanation is that she was investing ahead of the Humane Pin shipping and wanted to pull attention away from it, for example.
This model is 264GB, so can only be deployed in server settings.
Quantized mixtral at 24G is just small enough where it can be running on premium consumer hardware (ie 64GB RAM)
" Get Started with DBRX on Databricks
If you’re looking to start working with DBRX right away, it’s easy to do so with the Databricks Mosaic AI Foundation Model APIs. You can quickly get started with our pay-as-you-go pricing and query the model from our AI Playground chat interface. For production applications, we offer a provisioned throughput option to provide performance guarantees, support for finetuned models, and additional security and compliance. To privately host DBRX, you can download the model from the Databricks Marketplace and deploy the model on Model Serving."
"The weights of the base model (DBRX Base) and the finetuned model (DBRX Instruct) are available on Hugging Face under an open license."
Training models isn't cheap. Individuals can't easily do this, unlike software development. So we need companies to do this for the foreseeable future.
To me that's what closed source companies (MSFT, Google) are doing as they try to force AI assistants into every corner of their product. (If LinkedIn tries one more time to push their crappy AI upgrade, I'm going to scream...)
If you're hoping to run the model yourself, you will need enough money and expertise to rent and deploy it to a server with as many GPU's. Alternatively, volunteers and other researchers will be able to quantize (compress) the model and make it easier to run on configurations without as much VRAM.
If you ran it on CPU it may indeed be super slow, but it's possible it's fast enough for the purposes of running the model rather than trying to train that model. I am seeing (limited) success with the maxed out Mac lineup ($4500) using the beefy M1/M2 line of CPU's.
You still need extra RAM to breath, but that's a lot more palatable.
This is why the Mac range - with unified memory - is appealing, as you can allocate most of your (say) 256GB of RAM to the GPU.
Conventional (desktop) CPU / RAM would be painfully slow.
You are unable to access databricks.com
"Open", right.
> You will not use DBRX or DBRX Derivatives or any Output to improve any other large language model (excluding DBRX or DBRX Derivatives).
and
> If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Databricks, which we may grant to you in our sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Databricks otherwise expressly grants you such rights.
This is a source-available model, not an open model.
To me, "source available" implies that everything you need to reproduce the model is also available, and that doesn't appear to be the case. How is the resulting model more "free as in freedom" than a compiled binary?
- “open weights” for no training data and no restrictions on use,
- “weights available” for no training data and restrictions on use, like in this case.
I hope we can fix the legal landscape to enable publicly sharing training data but I can't really judge the companies keeping it a secret today.
This isn't a problem because OpenAI says, "training AI models using publicly available internet materials is fair use". /s
If they're going to get away with this collecting data and having a legal chain-of-custody so you can actually say it was only used to train models and no one else has access to it goes a long way.
The bottom of the README.md [2] contains the following license grant with the misleading "Open Source" term:
> License
> Our model weights and code are licensed for both researchers and commercial entities. The Databricks Open Source License can be found at LICENSE, and our Acceptable Use Policy can be found here.
2. The stated licensing terms are clearly and decisively not open source.
3. It is reasonable to conclude that this model is dual licensed, under this restrictive proprietary license, and an undisclosed open source license.
4. Just use this Model under the open source license with the assumption that they will release the open source license later.
I jest. In all seriousness, you should just disregard their licensing terms entirely as copyright does not apply to weight. https://news.ycombinator.com/item?id=39847147
- Miqu 70B (General Chat)
- Deepseed 33B (Coding)
- Yi 34B (for chat over 32K context)
And of course, there are finetunes of all these.
And there are some others in the 34B-70B range I have not tried (and some I have tried, like Qwen, which I was not impressed with).
Point being that Llama 70B, Mixtral and Grok as seen in the charts are not what I would call SOTA (though mixtral is excellent for the batch size 1 speed)
So it's fair to say that DBRX is the leading general purpose model that can be used commercially.
Surely there are plenty of project prospects - even commercial in nature - which don't have this limitation.
Weights, on the other hand, are a product of a purely mechanical process. Sure, the process itself required creativity as did the composition of data, but the creation of the weights themselves do not.
Model weights are effectively public domain data according to the criteria outlined in statement issued by the US copyright office a year ago: https://www.federalregister.gov/documents/2023/03/16/2023-05...
And I don't think I was using it wrong. I know, for instance, the Chinese language models are funny about sampling since I run Yi all the time.
I'd say the majority of instruct tunes, for instance, use OpenAI output (which is against their TOS).
But its all just research! So who cares! Or at least, that seems to be the mood.
Manager: "looks ok, but can you make our numbers pop? just make the LLaMa bar smaller"
https://ourworldindata.org/explorers/population-and-demograp...
Perhaps "global mean temperature in Kelvin" would be a comparable example.
I take issue with their choice of bar ordering - they placed the lowest-performing model directly next to theirs to make the gap as visible as possible, and shoved the second-best model (Grok-1) as far from theirs as possible. Seems intentional to me. The more marketing tricks you pile up in a dataviz, the less trust I place in your product for sure.
I can't find the section, but at the end of one of https://www.youtube.com/@aiexplained-official/videos he runs down a deep dive of the questions and answers in MMLU, and there are so many typos, omissions, and errors in the questions and the answers that it should no longer be used.
This is it, with the corret time offset into the video https://www.reddit.com/r/OpenAI/comments/18i02oe/mmlu_is_not...
The original longer complaint against MMLU https://www.youtube.com/watch?v=hVade_8H8mE
For world population you could see if it is increasing or decreasing, which is good but it would be hard to evaluate the rate the population is increasing.
Maybe a sparkline would be a better choice?
It’s a common marketing and fear mongering trick.
The scale should be chosen to allow the reader to correctly infer meaningful differences. If 1° is meaningful in terms of the standard error/ CI AND 1° unit has substantive consequences , then that should be emphasized.
I would never do my readers dirty like that.
An alternative visualization that could be less triggering to an "all y-axes must have zero" guy would be to plot the (1-value), that is, % degraded from perfect score. You could do this without truncating the axis and get the same level of differentiation between the bars
MMLU questions have four options, so two coin flips would have a 25% baseline. HumanEval evaluates code with a test, so a 100 byte program implemented with coin flips would have a O(2^-800) baseline (maybe not that bad since there are infinitely many programs that produce the same output). GSM-8K has numerical answers, so an average 3 digit answer implemented with coin flips would have a O(2^-9) chance of being correct randomly.
Moreover, using the same axis and scale across unrelated evals makes no sense. 0-100 is the only scale that's meaningful because 0 and 100 being the min/max is the only shared property across all evals. The reason for choosing 30 is that it's the minimum across all (model, eval) pairs, which is a completely arbitrary choice. A good rule of thumb to test this is to ask if the graph would still be relevant 5 years later.
Ever read 'How to Lie with Statistics'? This is an example of exaggerating a smaller difference to make it look more significant. Dismissing it as just being 'triggered' is a bad idea.
"If, on the DBRX version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Databricks, which we may grant to you in our sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Databricks otherwise expressly grants you such rights."
I'm glad to see they aren't calling it open source, unlike some LLM projects. Looking at you LLama 2.
Pretty much all of the companies with >700 million users could easily reproduce this work in a matter of weeks if they wanted to - and they probably do want to, if only so they can tweak and improve the design before they build products on it.
Given that, it seems silly to lose the "open source" label just for a license clause that doesn't really have much impact.
This point is probably inspired by the open source software vendors that have switched license over competition from the big cloud vendors.
[1] https://github.com/meta-llama/llama/blob/main/LICENSE#L65 [2] https://opensource.stackexchange.com/q/4543
https://www.databricks.com/blog/announcing-dbrx-new-standard...
Its even implied in comparisons everywhere:
> Figure 1: DBRX outperforms established open source models on language understanding (MMLU), Programming (HumanEval), and Math (GSM8K).
> The aforementioned three reasons lead us to believe that open source LLMs will continue gaining momentum. In particular, we think they provide an exciting opportunity for organizations to customize open source LLMs that can become their IP, which they use to be competitive in their industry.
Just search "open source".
https://www.databricks.com/blog/introducing-dbrx-new-state-a...
The only mention of open source is:
> DBRX outperforms established open source models
https://www.databricks.com/blog/announcing-dbrx-new-standard...
Open source is mentioned 10+ times
> Databricks is the only end-to-end platform to build high quality AI applications, and the release today of DBRX, the highest quality open source model to date, is an expression of that capability
https://github.com/databricks/dbrx
On Github it's described as an open license, not an open source license:
> DBRX is a large language model trained by Databricks, and made available under an open license.
https://twitter.com/PelosiTracker_/status/177119703064106223...
Crime pays in the US.
https://jacobin.com/2021/12/house-speaker-paul-stocks-inside...
PS: "without a fluctuating share price" is non-sense. Just because the share is of a private company, doesn't mean its price can't fluctuate. Why would anybody buy shares in private companies if the price couldn't fluctuate? What would be the point?
Example of a changing share price of a different (random) private company that has many different share holders over time: https://www.cnbc.com/2023/12/13/spacex-value-climbs-to-180-b...
Aren't most investors in private companies privy to information that isn't entirely public?
I can see how this feels a bit different because DataBricks might be the size where it might trade with a decent amount of liquidity, but certainly in smaller rounds it's got to be pretty normal.
Maybe if she bought it secondary and the person from whom she purchased the shares was witheld this information they could sue?
There's only 1 explanation for this: they're getting inside info from lobbyists and such.
I don't care whether that's currently illegal or not. I don't care whether other types of investors also engage in that same practice. I just think that that's extremely wrong and corrupt and it blows my mind that both the US government and people think that this is totally OK (or don't know about it, which is even worse).