ImprobableTruth

I'm curious, do you not think this might have adverse effects? Namely, if NN weights aren't copyrightable, limited releases like Meta has done might not be possible anymore so they might just cease completely with releases, ultimately leading access to large models to be more restricted.

>mmap is a really nifty feature of modern operating systems

... for some definition of "modern".

What does it look like if used context size increases?

I don't think there are any benchmarks for chat models. You could just do the usual lambada, etc., but what's the point? We already know the scores for llama and that RLHF doesn't meaningfully improve capabilities.

Is this a bit? If it's illegal to train on copyrighted material, then OAI has broken the law ten times over by training GPT3. There's absolutely zero reason for them to sue, they'll just ban the responsible people.

Open source models have been quickly following because most research has been happening in public. Who knows what will be if OAI, Google and Deepmind all stop publishing their results?

"desert" means something one deserves. You probably know it in the idiom "just deserts".

Big Tech on its own will already push this technology very far and they don't give a damn about safety, only the optics of it.

I'm not convinced that small actors will do much damage even if they access to capable models. I do think there's at least the possibility that essential safety work will arise from this.

OpenAI has been giving this as their reason for years and nothing has happened even though powerful models were released open source.

Turns out that misinformation is already incredibly cheap to create and the bottleneck lies elsewhere.

ChatGPT has over 100 million users 2 months in. It's not even remotely comparable to VR or blockchain.

Distillation would be the ideal way (especially because it also has efficiency gains), but as far as I know distillation for LLMs is kinda unproven.

Honestly though, even if you just finetune it, which you will want anyway for any serious commercial application, it's essentially impossible to determine the origin.

The training code is not available.

I don't want to be too cynical, but OpenAI used to be more open too until they decided releasing weights was too dangerous (/not profitable enough?), what guarantee is there that Eleuther doesn't also close their doors at some point?

It's tokens processed, not generated.

They offer an open base model and then offer fine tuning to companies e.g. apparently they're creating finetuned models for movie companies.

GP contrasts DP and DDP by saying that DP is "where you clone your model over each GPU" and DDP is "'proper' multi-GPU training - you can now train big models and put a little bit of data on each GPU". That's simply not what DP or DDP is. What could this possibly mean if it's not misunderstanding DP as data parallelism and DDP as model parallelism? I'm fairly certain that what they're describing is using DDP (which only does data parallelism) in addition to model parallelism.

I phrased that wrongly, DDP itself doesn't of course. I meant that using it in the way GP does is also doing model parallelism.

DistributedDataParallel (potentially) does both model and data parallelism. Data parallelism is also absolutely used when training large models, it has its downsides, but I don't think there's any way around it if you're training with a large amount of gpus.

Unfortunately non-commercial and only available to academics upon request.

Calling this garbage is absolutely wild. The authors make it very clear that this is optimized for throughput and not latency. Throughput focused scenarios absolutely do exist, editorializing this as "running large language models like ChatGPT" and focusing on chatbot applications is the fault of HN.

It's also a neat result that fp4 quantization doesn't cause much issue even at 175b, though that kinda was to be expected.

How fast is it in single batch mode?

No, because they're already taking that into account.

>Metric: generation throughput (token/s) = number of the generated tokens / (time for processing prompts + time for generation).

(Though they're doing batching, so this is an unfair comparison. Would be interesting to get single batch speed.)

You've made one huge mistake: Davinci's $0.02 is not just per 1k tokens generated but also context tokens consumed. So if you generate 50 tokens per request with 1k context, the price is actually 20 times as large at $0.40 per 1k tokens generated - much less palatable, costing 3 times as much as the cloud hosted version of this.

And that's not even taking into account the gigantic markup cloud services have.

Controlnets are just a clever method of finetuning a network to enable additional conditioning. Nothing about it is specific to latent diffusion, so why would another approach like pixel space diffusion not work with it?

Also, I'd hold my horses with calling latent diffusion "out of date". The Imagen paper notes that the results greatly scaled with text-encoder size and the model DeepFloyd (it's a team) are working on uses T5-XXL (the unet is also a decent bit larger, but the Imagen paper claims that this should have a much smaller influence). Latent diffusion might also be able to work with text if you scale up the text encoder.

I'm very skeptical of Google's "trust me, I have the hottest model, it just goes to another school". Muse might have nice fidelity and be fast, but the example images have worse visual quality than Imagen. I definitely wouldn't be surprised if transformers took over, but I don't think it's a foregone conclusion.

It's nice, but a far cry from gpt-3

"Stochastic parrot" just strikes me a typical motte-and-bailey. The narrow interpretation ("LLMs only learns simple statistical patterns") is obviously wrong given the capabilities of ChatGPT, while the broad interpretation ("LLMs are text predictors") is trivial and says nothing of worth.

I've heard this often, but I'm not convinced. C just lacks mechanisms to accommodate something like generic data structures, all the libraries I've found feel very hacky, it's either void pointer or macro overload.

Honestly, static typing encouraging data shape documentation alone makes it worth it to me. Yes, they're pretty hacky in Python and don't feel great, but I'm sure it'll continue to improve.

I'm just so over having to guess what a function might accept or return. Life is too short to spend it constantly reverse engineering code because people can't be arsed to write proper documentation.

What part specifically? Java is (in)famous for its orms after all. Unless you want to count that as clunky codegen? I'd argue that all metaprogramming boils down to that though.

Which of these do you think require dynamic typing?