Comment by ianbutler | Hacker News Reader

Comment by ianbutler | Hacker News Reader

I think this is more of an issue of smaller models with larger context aren't a magic bullet and I'm not sure anyone really paying attention thought they would be.

Falcon 40BN, after extensive use, is the only model that has crossed the "I'd build a product on this" threshold in open source. I expect my tune won't change there until we see larger Open Source models trained on high quality data. These are the models that will benefit from the extended context lengths.

But longer context without a model capable of utilizing the context to actually do things, is yeah, unsurprisingly, not useful.

There's also not really an incentive to currently make that happen in Open Source. If you can train a big model, with a unique high quality dataset you've assembled, using some novel techniques on some performant serving infrastructure, you basically have a blank check right now.

Edit: On my last point, I'm going to say I was being very jaded. I think the community is doing a lot of awesome things and I know of a few efforts personally to train these larger models for open source consumption. I don't mean to be so overly negative here.

It's pretty amazing to me that there are any open source models that have crossed that threshold - especially given that prior to the Llama release in March there were hardly any open source models that felt like they could catch up with GPT-3 at all.

If you're looking for an incentive to make this happen in open source, how about the team behind MPT-30B just getting acquired for $1.3 billion.

There are a LOT of well-funded groups trying to make an impact in open source models right now.

I'm highly skeptical that people are just going to leave the secret sauce out in the open for anyone to directly compete with them, feels a bit too idealistic to me.

If you look at Mosaic, their value, their secret sauce, is the infrastructure. The model, to say it cynically, was marketing. Now it's true it can be marketing and help push open source forward at the same time.

More of a separate point, but I also think models are going to be fairly quickly commoditized on a couple years timescale, so the infrastructure to train a model or serve a model, or the thing built on the model is the value, but right now models are still the value because they're so new and we haven't finished (in fact we're really only starting) the race to the bottom.

Sure, OpenAI and Anthropic clearly have some secret sauce that they're holding back... but the rate at which the open source research community has been catching up is quite frankly astonishing.

I would be very surprised if the closed source research teams can maintain their advantage over the next 12-24 months - especially given the number of enormous GPU clusters that are becoming available now.

I don't want to diminish the efforts of the community, I agree the work is enormous and the strides are pretty crazy. I honestly hope I'm wrong. I was talking with a friend and discussing that this is one of the coolest collaborative periods I've seen in tech so far. Hopefully it plays out your way and not mine :)

What is secret about infrastructure?

I’ve been testing the MPT models and they’re better than all the other open source stuff I’ve tried. I’m using my task-based, one-shot prompts that are about 1-2K tokens. Seems pretty on par with GPT-3.5. Marketing or whatever, first time I’ve been at all impressed.

Falcon was trained by a research think tank, and this is going to happen more and more as AI compute gets cheaper and the frameworks get better. We aren't going to be stuck with overpriced A100s running early PyTorch eager mode code for long.

High quality "open" data is an tricky problem, but the community has already come up with some impressive datasets for finetunes, like the roleplaying collection used for Chronos and Pygmalion. I think this is going to get better too, epecially if cheap models can "distill" lower quality human datasets.

> We aren't going to be stuck with overpriced A100s running early PyTorch eager mode code for long.

I agree and am eagerly awaiting this on the hardware side and trying to participate in this where I can on the software side.

> Falcon was trained by a research think tank

A research think tank subsidized by UAE’s oil wealth. Somebody get Masayoshi on the line, the Saudis must be hungry for deals for their vision fund.

Unnecessary ad hominem. Every research tank is subsidized by someone.

Very true, MIT should disclose their military affiliations too when publishing. Otherwise, they are just dishonest

They do. The grants themselves have requirements for acknowledging the funder in publications; most researchers also mention the funding sources for their work and projects on their website and in their CVs.

People who get those grants are proud of the fact and like the money.

Schmidhuber, the father of deep learning, is already on it: https://cemse.kaust.edu.sa/people/person/jurgen-schmidhuber .

"The father of deep learning", hah. Sounds like something he would say.

The phi-1 paper suggests small models trained on higher quality data hold promise as well. If confirmed that would be great news for the potential accesibility of training open source models.

https://arxiv.org/abs/2306.11644

Doesn't this imply some kind of scaling law? An LLM of complexity Y will only benefit from a context length of at most Z? That feels truthy to me.

It dependends on the context (hah).

For, say, continuing a story, even a small LLM will benefit from a mega context even if it struggles to retrieve specific information... As long as perplexity isnt hit too much, which depends on the method.

But this is more of an issue for summarization or information retrieval.

Really? I found falcon 40b underwhelming compared to llama 65b

Try falcon-40b-instruct, that’s optimized for better interaction with instructions and performs A LOT better

Reply on news.ycombinator.com