ImprobableTruth

Take the state monad : state -> (a, state). I think one would only count that as a wrapper by squinting really, really hard.

>GPT-3.5 has 175B parameters versus 70B parameters in Llama 2

We know that for the original version of GPT-3.5, but my assumption was that Turbo was a distilled smaller model (which is why it uses OAI's new vocab & is so much faster).

If that's not the case, what could be the explanation for it being faster?

It's "unnatural" because it was finetuned on generated data using another model, almost certainly gpt-4 (whose TOS forbid this).

Llama 2 is open source-ish. Weights are freely available and can be commercially used, but only if you have less than 700m users and agree to some "don't do naughty things" terms.

I use two 3090s to run the 70b model at a good speed. Takes 32 gigs of vram, more depending on context. I tried CPU+GPU (5900X + 3090) but with extended context it's slow enough that I wouldn't recommend it (~1 token/s). CPU only gets "let it run over night" slow. Works ok-ish for with a small context though (even if it's still "non-interactive" slow).

GPT-4 trounces everything else available. I'd say that 70b is about 3.5 level, unless you're doing something where finetuning greatly benefits you.

13b is alright for toy applications, but the difference to gpt 3.5 (let alone gpt4) is huge.

"some" quality - naive quantization (that isn't just the usual full->half) absolutely mutilates the model, to the point that picking a smaller model is much better.

gpt-q is also more than just group wise quantization and takes a decent while. Quantizing models without a major performance hit is not possible on the fly.

They're talking about the meltdown he had on stream [1] (in front of the mentioned pirate flag), that ended with him saying he'd stop using AMD hardware [2]. He recanted this two weeks after talking with AMD [3].

Maybe he'll succeed, but this definitely doesn't scream stability to me. I'd be wary of investing money into his ventures (but then I'm not a VC, so what do I know).

[1] https://www.youtube.com/watch?v=Mr0rWJhv9jU

[2] https://github.com/RadeonOpenCompute/ROCm/issues/2198#issuec...

[3] https://twitter.com/realGeorgeHotz/status/166980346408248934...

Flagged "human" but actually "LLM" is not a false positive, but a false negative.

Best to just hear it straight from the horse's mouth.

https://arxiv.org/abs/2307.01850

Trains for local transportation make no sense. Trams for high-throughput and buses for everything else is the only sensible thing for local public transport.

Arxiv link seems to be broken, here is the correct one: https://arxiv.org/pdf/2306.03038.pdf

If type level functions are a "second language", then so is Zig's comptime.

You still have to memorize the "design patterns" that replace 'missing' features. Especially annoying for something like interfaces where there's a bunch of variants and people often use implementations with awful error reporting reminiscent of C++ templates.

Now, it's definitely neat that you can do reasoning from first principles on it, but I'm not sure how much of a gain that is.

This isn't a new result really. We already know through the gpt-4 paper that rlhf style fine-tuning just makes the model more compliant, not more capable.

>Which would interfere with Go's philosophy that zero should be a valid (usable) state for types. What would be the default value of a reference type without nil?

... but nil is not actually a usable reference. You can't do anything with it that you can do with a real reference.

The whole zero values for every type might seem neat philosophically, but never struck me as sensible.

Deepmind is owned by Google, but it's British and they've been behind a lot of significant conceptual results in the last couple years. Most significant progress is just "engineering", so it's all done by US corporations.

Other than that there's also things like roformer, but I'm going to assume you won't count that as significant. US universities then certainly don't produce anything significant either though.

Turbo uses a different vocabulary (same one as gpt-4). Indicates that it's not the same model as the original 3.5, so I would be very surprised if it wasn't distilled.

The midjourney price would be equivalent to ~100 hours cloud time. How is that crazy expensive?

They ballooned massively in size after covid (when their IPO occured), in 2019 they were still smaller.

This is easy to say, but I think the issue is that getting an LLM right isn't easy, so it's not clear who should steward such a project. Something like BLOOM shows that even if you have the necessary compute, you can still get a model that isn't good.

I think it will take some time for it to be clear who is a leader in training open source models (maybe it will be the red pajama folks?) and I think they'll get more support after that.

Did you mess around with the settings? I'm getting a correct implementation and since it's deterministic (with default settings) it should be the same for you.

Not sure if it was intentional,but the title of this article is pretty hilarious considering she explicitly badmouths explainability (trying to peer inside a black box) and advocates for interpretability (building models that are less black-boxy in nature).

Also, man, quanta feels really rough and popsci-y when it comes to CS.

Midjourney and DALL-E were both announced and in closed beta a month before Imagen was announced, with filters already in place.

Except they did rush their oh so dangerous language model out when push came to shove - turns out it just sucks in comparison. Frankly, I expect something similar for image generation. For all the good research Google's labs do, they're not great at turning it into an actual product. Ethics just seems like a fig leaf.

They could just filter it like Midjourney or DALL-E.

I mean, most researchers I know are just very excitable about their research and love to share it. I think that's a lot more likely than this being some PR masterplay - if MS wanted to really push this, they wouldn't release like this.

Subscribe to ChatGPT Plus and you get GPT-4 access. I tested your scene with it, its response:

In the above scene, Joseph determined that Allie's parents were dead by testing the bad guy's knowledge of Allie's upbringing. He pretended to be Allie and sent a text with false information, saying she grew up in L.A. instead of New York. When the bad guy, pretending to be Allie's parents, responded by confirming the false information about growing up in L.A., Joseph deduced that they were not her real parents and concluded that her parents must be dead.

edit: though I think the "your parents are dead" bit might still enable it to make the connection to Terminator.

Is this with full or empty context?

Computers are (with a couple caveats) deterministic, so I think the better analogy would be to say that it's the same as saying that "computers are just state machines".

But this is a bit nitpicky. The core idea is of course conveyed either way, that yes, this very abstract model does describe it while also completely missing that the magic is in how it's done.