Meta rolls out AI language model LLaMA | Hacker News Reader

Meta rolls out AI language model LLaMA

reuters.com

124 pointsthm100 comments

Open articleView on HN

It's a funny accident of history that one company's business model in 2023 is a combination of:

1. Large interactive language model

2. Wireless helmet for 3D equivalent of conference calls

3. Helping you keep track of which classmate from high school went bald

4. Birthday reminder service

5. Virtual currency intended to compete with the dollar

6. Running all school and church message boards in the US

7. World's largest collection of food photography

8. Instant messaging

Ever heard of Facebook marketplace?

Besides every mom and pop shop uses FB pages and groups these days. With mobile carriers supporting free bandwidth for FB it's basically part of infrastructure in many 3rd world countries.

You are completely right; I should have known I'd forget about entire lines of business making that list.

They dropped the currency idea.

https://ai.facebook.com/blog/large-language-model-llama-meta...

Even with this it's a little unclear, is this open source?

What kind of machine is needed to run it?

> Access to the model will be granted on a case-by-case basis to academic researchers. Non commercial license

Microsoft and OpenAI, Google losing $100bn from a bad demo, layoffs, cost cutting pressure to innovate again... I feel the days of full openness in AI research from corporations are over.

I think we indeed hit "peak open-source" for AI and there unfortunately won't be as much sharing in the coming years. When the economy is down, people and companies think more in "zero-sum-game". I hope to be proven wrong.

https://twitter.com/EMostaque/status/1629160706368053249

This is what everyone needs to watch. It was Stability.ai that spooked OpenAI for disrupting DALLE-2 with a open-source AI model; Stable Diffusion. I am betting they are going to do it again for ChatGPT.

The endgame for this AI race is obvious and when it comes to AI models, open-source ones always disrupt fully closed source AI companies.

But first, we'll see which 'AI companies' will survive the lawsuits, regulations and fierce competition for funding.

The golden age of open-source AI is ahead of us. Open-source AI companies are being launched and funded. High quality, large, labeled data sets have never been more accessible, and scaling law plateaus means there is going to be a lot more momentum on data- and compute-optimization, meaning current SOTA models will start fitting on smaller and smaller hardware, down to commodity hardware.

Until legislation clamps it down.

At least in the United States, it's well established that code is protected by freedom of speech. https://www.eff.org/press/archives/2008/04/21-40

Yeah I don't expect that to last very long. Once it is discovered you can have an AI army that will tear apart any human force like tissue paper, AI will be classified and regulated as a WMD. They'll amend the Constitution if they have to, or get SCOTUS to do it for them.

Until recently, reproductive rights were well established too. Shit changes, yo.

This is why activist judiciaries inventing rights that plainly are not there (and in fact, supporters at the time understood that this was on shaky ground legally) are dangerous to rely on.

The situation here is not comparable.

This. We already had discussions of 'attacks' on models based on public data sets so 'good sets' may soon become the thing to go after ( and suddenly data brokers may really want to up the prices of their sets ). We might actually see more privacy as a result as the data brokers will start charging a premium for clear sets.

Naturally, as predictions go, don't quote me on that. I was wrong before.

There is hope: https://www.bloomberg.com/news/articles/2023-02-21/amazon-s-....

I wouldn't be remotely so quick to throw in the towel. ML research tends to operate in "jumps" and plateaus. At this point, the concepts behind the big LLMs are relatively well known and the bottleneck is cost of compute + cost of training data. Thing is, cost of compute keeps coming down.

OpenAI's "win" wasn't even so much in the research but in the design of ChatGPT as an interface. Its own model makes the same kinds of egregious mistakes as google and FB's own LLMs. Also, OpenAI was willing to just deal with the ethical fallout of releasing it into the wild with the ability to generate authoritative sounding falsehoods.

I suspect we're going to go back to a period soon where a lot of the innovation we're seeing is around interfaces and infra to make interacting with LLMs natural and applying them to product use cases where they make sense.

I really don't blame OpenAI for ethical issues over opening up access to ChatGPT. They're not claiming it's responses are factually correct, and arguably by making it openly available they have done more than anyone else to raise awareness of the risks and limitations of LLMs. We need access to these things to make informed decisions of what are or are not appropriate uses.

Microsoft and Google are a different story, they're specifically pushing these as authoritative sources of information. If we hadn't had access to ChatGPT and the ability to learn it's ins and outs, it might have taken longer to expose so may of the flaws in the Microsoft and Google services.

What are they claiming regarding the correctness of the responses?

The OpenAI disclaimer is pretty comprehensive. Basically nothing.

why does Yann LeCun advertise that it is open source then?

I assume you could train one like this provided that you bring your own training data and computing resources.

Because it is open source: https://github.com/facebookresearch/llama

It feels like open source as a term became obsolete(-ish). Is it open source (in spirit) if only the inference code is open source? I mean, technically it might be, but not being able to train it myself is basically against FSF's freedom 1 if you consider the model the software.

That repo is basically releasing a binary (the weights) and an open source runtime to run them.

I agree… This is not open source. If you publish a wrapper that represent just 0.01% of your code and your product and hide all the rest as closed source that is considered open source now? Then any closed source providing an API become open source with this bogus definition. I guess now Windows is open source too, just miss the 99.99% of the rest of the code.

It’s absurd and it damages the open source world and its real definition.

No, this is open source. The model code and everything needed to verify their results is provided. No one promised the training code, and the training code is not needed to reproduce the results.

Going back to my original comment: if I release a giant binary blob (say, a Windows image) and an open source runtime to run it (say, VirtualBox), does that mean I can say Windows is open source? I don't think so, and to me it seems that this situation is a perfect parallel.

This is FSF's freedom 1: "The freedom to study how the program works, and change it so it does your computing as you wish (freedom 1). Access to the source code is a precondition for this."

By that definition I don't think you can say this model is open source then.

You could say the inference code is open source, and that's technically true (and the only thing the repo claims TBH), but calling this open source AI model is misleading.

Open source inference code is outright uninteresting. It's a few hundred lines of glue code to run a giant binary blob which does the heavy lifting, i.e. the actual computing, i.e. the actual software, i.e. the software whose source we're actually interested in.

EDIT: Actually I just noticed they didn't even open source checkpoints+tokenizer.

Did you even look at the repo before commenting? The model code is literally open source: https://github.com/facebookresearch/llama/blob/main/llama/mo...

Even the checkpoints are provided - for free! All you have to do is ask.

Someone at Facebook spent a ton of money to train a state of the art model, open sourced the code and even provides checkpoints free of charge, and you still complain? The level of entitlement is off the charts…

> All you have to do is ask.

That makes it literally not open source. Can I redistribute it freely? I don't think so, I might be wrong. If not, it's not open source.

> and you still complain?

But I don't complain about Facebook releasing it in whichever terms they want. I complain that people call it open source when it clearly isn't.

> The level of entitlement is off the charts…

How is pointing that it's not open source entitlement?

Again, I have to point you to the repo code. Here’s the licence: https://github.com/facebookresearch/llama/blob/main/LICENSE

The model code is literally open source. Provided and licensed.

The disconnect here is that the model code isn’t the important part. If you read the paper, LLaMA is all about the training data and the training process (which required a ”massive quantity” of compute, to quote the paper). Without releasing the significant part of their work, what are they really releasing?

Exactly - massive amount of compute required to reproduce the training process, and that’s why almost no one is going to try training this model from scratch. Most people will use this model as is, and a small number of people will finetune it on their own data - those people most likely already have their own training scripts, because finetuning is much easier than training from scratch and requires different hyper parameters.

I don’t know why they haven’t released the training code - I agree it would be nice if they did - but the important thing here is they created something valuable, open sourced it, and even provided model weights - for free. Let’s appreciate it.

Even if they did publish their training code - that’s not enough to train the model. You also need their dataset. Would you still claim “the model is not open source” because the dataset is not available?

Bottom line: the model has been open sourced. The training code hasn’t - and isn’t needed by most users of this model.

> You also need their dataset.

With the training code you can fine tune the model with your own dataset, which exactly fits FSF's freedom 1.

>I might be wrong. If not, it's not open source.

It isn't "Free Software" if it doesn't allow for the four freedoms -including the right to modify and redistribute your modifications.

As a term, "open source" is ambiguous to almost having almost no actual meaning at all. But "Free Software" has a concrete meaning and if you can't redistribute something then it is not "Free Software".

Hope that helps.

Edit to add a citation: https://fsfe.org/freesoftware/

I know what free software is. I meant the checkpoint might be redistributable after you ask for it (but I doubt so). Hard to tell since you have to ask to get it, but I'm pretty sure you'll get it with a pretty restrictive license.

Open Source has a clear and specific definition BTW: https://opensource.org/osd/

Would you say that Doom is not open source because the WAD are not freely available?

That's what I meant when I said the open source term kinda got obsolete. With AI the model is the program. I don't even have the training code or the checkpoint so I cannot modify it as the freedom 1 requires.

As I see it the weights are more similar to LLVM IR. It got "programmed" by gradient descent.

At most you can say the inference code is open source (see sibling thread).

The confusion here is about what constitutes a “model” - a model source code, model parameters, code to train the model, dataset used to train the model, code used to evaluate the performance of the model? I would probably say that the first two constitute a model. I agree that only the first one (model code) is strictly open source, while the second (model parameters) is not, because it’s not available to everyone.

Here's hoping someone with access puts it on bittorrent.

DLP, overall architecture, and many more aspects put the kibosh on that.

The career risk isn't worth it especially when tech is deployed client- and network-side to detect just such exfil attempts. The average of network, security, and client management staff tend to be PEs (SREs) who can code, some have PhDs, and are the cream of what was previously organized as "corporate IT" world. So I fail to see any incentive to throw away their career and reputation by giving away IP for $0.

There's a metric s*ton of optimized hardware to generate models. And I have my doubts if Sama at OpenAI, even with 10 gigabucks from Microsoft, can sustain growth, organizational culture, and long-term investment at the scale others are bringing online with less trouble and more experience.

The future interaction will AI models will most likely be through an API because the models themselves are becoming too large to fit even on the most extreme DIY NAS solutions.

TL;DR: it's not happening.

> The future interaction will AI models will most likely be through an API because the models themselves are becoming too large to fit even on the most extreme DIY NAS solutions.

And yet I spent last night running GPT-J-6B on my desktop CPU at 2 tokens / sec. People are finally starting to optimize these models, and there's a ton of optimization to go. We'll definitely be running these locally in the next few years. This model especially looks like an ideal candidate for CPU optimization, given the pairity with GPT3, and that it's within spitting distance of the size of models like GPT-J-6B.

There are plenty low paid students or working students or admins in-between.

You can make small LLMs more performant than large ones like GPT-3 by fine-tuning on specific tasks or providing them tools to offload precise calculations:

e.g. Toolformer: https://arxiv.org/abs/2302.04761

which uses APIs and functions to improve GPT-J beyond GPT-3 for various tasks

How dare you bring knowledge in here! Some people at work would be offended if they couldn't buy their A100 or DGX toys, or had to scale back their custom ASIC and systems R&D.

Once it is out there it can be rehosted on the clear net since model weights can't be copyrighted (unless they intentionally over train on some copyrighted snippets they own maybe).

BitTorrent links: https://github.com/facebookresearch/llama/pull/73/files

> Meta's LLaMA, short for Large Language Model Meta AI

I would have said "LLaMA, an acronym based on cramming concepts together until the word Llama is suggested, and then as a flourish, adding an unnecessary lower-case letter 'a' in the middle. Llamas are unrelated to AI, so this should not have been anybody's priority." Guess that's why I'm not a writer.

As long as it is pronounceable, memorable, it's actually a good one.

Also known as a "Backronyn".

Start with the acronym you want, and work back from there..

At least it's better than LION (evoLved sIgn mOmeNtum).

https://arxiv.org/abs/2302.06675

That's a cool acronym, if only for the novelty.

Very cool to see such model weights getting released. Makes me wonder, seeing that Meta is crushing Google... This is like MySpace teaching a lesson to Apple.

>seeing that Meta is crushing Google... This is like MySpace teaching a lesson to Apple.

Huh? How so? I see Meta and Alphabet as very similar companies. Both rely on ads, extracting information from text, keeping users engaged on their platforms, furthering AI/machine learning, etcetera. Does MySpace have their own computing devices I'm unaware of?

But they are not released. Only for non-commercial use by groups that they grant access to. Which means studies funded by giant corporations, or governments using it for their own propaganda, or hackers that steal it or find a leak and use it for spam and phishing etc. Anyone who wants to make something useful and positive is going to be left out.

AI companies seem to be in a paradoxical space where they need the optics of being open even if it goes against their business interests.

You know, Google releasing a ChatGPT competitor that could be run locally would really take the wind out of Microsoft's sails without needing to shift their own business model.

Yan LeCunn seems to be parading that this is open source under GPL V3, which would have been great. However, yes the model weights have not been released and are under a separate non commercial license.

"Researchers and entities affiliated with government, civil society, and academia" sounds like it might include some positive uses, if you don't assume Meta is maximally evil just for the fun of it.

I don't see any crushing going on.

Related discussion: https://news.ycombinator.com/item?id=34925944

the level of the comments in this post is disappointing for hacker news.

Absolutely. From what I can tell, this model hits state of the art on several benchmarks at 1/10 the size of its benchmark winning competitors. That performance efficiency is great to see because while we know we can increase performance by scaling up the size and compute requirements, getting the same performance out of a fraction of the compute is a major win. This will be especially valuable as these models begin to see real production use at scale such as with ChatGPT and Bing.

HN seems to have mostly devolved into skepticism, hype and confusion lately around all things AI. I’d say we have just hit futureshock.

How can they avoid that someone perform a post-training from this model to obtain a different model that perhaps could not be protected by copyright?

the focus on non-proprietary training datasets and computational efficiency for the purposes of reducing costs and democratizing access is pretty cool!

it is interesting how model performance seems to scale almost linearly with size for the sizes chosen. (or perhaps not, perhaps the researchers are choosing to focus on the part of the performance-size response curve that is linear).

The language model that lies better than any other model in the universe? (just guessing from the training data)

Is there something special about this that would motivate someone to bother when they can access alternatives with less restrictions / more openly?

The model has less technical debt from being newer. The quality per cost is supposed to be good.

The name made me think of this: https://youtu.be/KMYN4djSq7o

It made me think of Winamp.

ah, I expected that to be this: https://www.youtube.com/watch?v=kZUPCB9533Y

Or this: https://www.youtube.com/watch?v=gTmEfoPnQDo

Better than LMAA I guess O:) https://en.wikipedia.org/wiki/Swabian_salute

There will one day be a wikipedia equivalent organization that maintains a language model.

It really whips…

dumb question - but - can we actually use this right now? Like download the model and run it locally?

Yes, everyone in the business of luring suckers will grow themselves one of these.

All of this just to stay relevant among the competition, and yet, none of these AI giants has introduced something that just works.

Waiting for Apple.

What makes you think that apple can even compete at all? The businesses they are competing with in this space have actual monetary incentive to do research in the field, this sounds like the comments people make about apples electric cars and AR headsets that are always just around the corner and better than anything that already exists

That's fair. Apple does work on all kinds of stuff that never makes the light of day, mainly because it shouldn't. It's not better than the competition until it ships and proves it in the market. Apple have their misses too.

I'm not convinced Apple needs to, or is interested in competing in this space. Yes they have Siri, but there's no guarantee these language models will ever actually be safely functional in a role like that.

Even if they are, is this really an interface people will want to use? I did use Siri to set alarms and such for a while, but even though it did it reliably and conveniently, eventually I stoped bothering. Who want's to talk to their phone on the train? I just found it cognitively less intrusive to just set alarms manually.

I wouldn't trust this with a 10 foot pole. Given facebooks record why would we as a people's even allow this?

I don't trust Facebook at all, but we need more than just "THE EVIL AI WAS MADE BY AN EVIL COMPANY" what is the concern? They're going to shove ads into your AI?

They impermissibly used customer's information in the data set? that they knowingly disregarded known risks of harm associated with the product?

got any citations on that one, mate?

I didn't make any claims, someone asked what could facebook do that would be of concern, and I gave two examples of their past conduct.

I would hope not, have you seen the dumb stuff my relatives say? I think you need to provide some evidence to back your claim though.

I didn't make any claims - someone asked what could they have done, and I listed two things they did recently...

> https://research.facebook.com/file/1574548786327032/LLaMA--O...

> Dataset Sampling prop. Epochs Disk size

> CommonCrawl 67.0% 1.10 3.3 TB

> C4 15.0% 1.06 783 GB

> Github 4.5% 0.64 328 GB

> Wikipedia 4.5% 2.45 83 GB

> Books 4.5% 2.23 85 GB

> ArXiv 2.5% 1.06 92 GB

> StackExchange 2.0% 1.03 78 GB

At no point do they use information from Facebook customers

No, we don't anymore than that. Would you trust a fucking murderer?

Facebook has been weaponized numerous times, why would this be any different? Why would they be given any benefit of the doubt?

Facebook has a record of tainting their machine learning models? I was not aware of that.

It is disingenuous to pretend not to understand that unethical behaviour in one area is a strong predictor of unethhical behaviour in another area.

It's disingenuous to make assumptions and argue without having made a good faith effort to see if what you're saying is true. 5 minutes of research would have taken you to the research paper in question, which has a straightforward description of the training data sources:

CommonCrawl 67.0% C4 15.0% Github 4.5% Wikipedia 4.5% Books 4.5% ArXiv 2.5% StackExchange 2.0%

So in other words, nothing here in this research has used people's social media posts, and there's no private data whatsoever because the sources used were all public.

Good to know, and I always appreciate and respect people bringing statistics and citations - but this in no way refutes my core point that it is always reasonable to assume that Facebook will act unethically based on their past behaviour.

You've provided a strong datapoint indicating that the training sources in question were ethically sourced* - that's good! I'm glad to hear that Facebook avoided _this specific way of acting unethically_. This does not affect in any way my judgement of the likelihood of Facebook acting in _other_ unethical ways in this area, such as using the collected data in unethical ways.

* I don't, personally, hold the opinion that "just because something was posted publicly means that it is ethically permissible to do _anything_ with that content", since it's unreasonable to believe that the average internet user is giving informed consent to uses like this - but I do concede that that's a more subjective perspective and is probably not supported by the letter of the law. Luckily, this isn't relevant to my argument.

As an aside - I clicked around in a few pages on CommonCrawl's site (https://commoncrawl.org/big-picture/what-we-do/, https://commoncrawl.org/big-picture/frequently-asked-questio..., https://commoncrawl.org/the-data/get-started/) and didn't see any statement to the effect that social media posts were excluded, and a Google search for "Does CommonCrawl index social media" didn't provide a definitive answer, either. You're under no obligation to prove it to me - and, as I detailed above, even if you did, it wouldn't affect my general distrust of Facebook.

It's disingenuous and frankly intellectually cowardly to continue giving a bad actor such as facebook benefit of doubt. It reeks of enlightened centrism.

What in facebooks past gives anyone reason to believe this model would be used for anything other than nefarious purposes?

I’m biased because I used to work for Facebook a long time ago. So take my words with a grain of salt.

But Facebook research really wants to build amazing tech to power new experiences. The company has attracted many smart and amazing developers who want to pioneer innovation and want to see their work thrive in the broader tech community.

I’m sure this will have the same issues as the other Large Language Models and there’s a lot of work to be done to figure out how to distill a reliably useful system from something trained on so much human language which is unreliable.

I’m heartened to see them talk about needing to research how to remove bias, toxicity, disinformation and hallucinations.

This is something we need to focus on for all LLMs so I’m happy to see more focus on that in the community.

The FAIR group is top notch.

I don't really think it matters what the research team wants or doesn't want. That is totally insignificant to what the decisions makers want and implement into the facebook product.

What you helped work for was to build a giant spy tool for corporate and government actors. Your own friends, neighbors, and even family surely have some kind of shadow profile on facebook, sleazily tied to other services, packaged, and sold on equally sleazy and opaque data markets.

Personally, I would support the government just shutting down facebook altogether, by force if necessary. They have lost all trust with me, and any benefit to society has easily been drowned by their numerous public failures, up to and including genocide.

Like, once your company is involved in genocide, that should be it. Game over.

Reply on news.ycombinator.com