brrrrrm

should really be titled streaming output, as full duplex streaming isn't mentioned at all. that'd be necessary for things low latency things like speech etc.

goes to show formal language isn't a necessary component of high communication. might even be antagonistic

It kinda has already with fixed matrix multiplication units. But beyond that, no chance. Bitcoin is an unchanging hash algo, not a developing software

If you're hitting a memory wall it means you're not scaling. This stuff really doesn't apply to scaled up inference but rather local small batch execution

perhaps the title should be "Llama2-70B worse than humans ..."

> actually conducting research

5 people evaluating 45 responses? This doesn't take a year to do. The issue is that this study is was poorly funded and slow - model development has far outpaced the results and there's likely little to no longevity in the outcome.

> The most promising model, Meta’s open source model Llama2-70B

This is an old model that was not dominant even when released. This study must be fairly old or I question the qualification of the group running it.

It’s all about the kernels tho. The language doesn’t matter much. For the things that matter, everything is a dispatch to some cuda graph

I’m not really a fan of this convergence but the old school imperative CPU way of thinking about things is dead in this space

not really. embeddings are embeddings. check out llava

LLMs can deal with more than text. Impressive today is nothing tomorrow

we've like barely trained these things?

the entirety of common crawl is 424 terabytes. that's merely 6 days of 8K raw video.

So, noticing that linearized models have tiny KV caches ahem i mean state spaces, this approach tries to increase their size along the embedding dimension. Increasing this enormously by applying a different softmax (which is compatible with the expanding tensor product) yields a very symmetric mathematical structure that can be exploited to recover some efficiency.

Is that right?

how does this approach differ from Nvidia's 2019 writeup on using trees to improve allreduce operations? https://developer.nvidia.com/blog/massively-scale-deep-learn...

this is true of even just matrix multiplication (A*B) of which attention has two

For most LLM workloads today (short text chats), hundreds or a couple thousand tokens suffice. attention mechanisms don’t dominate (< 30% compute). But as the modalities inevitably grow, work in attention approximation/compression is going to be paramount.

Nice to see Pytorch already elegantly supporting this next step in research

amazingly, the browser has pretty much solved all of this. fully compatible EMCAscript implementations on every single device with hardware access (such as the camera, as is needed in this post)

I don't buy your "security through difficulty" argument for that reason alone.

Empowered with chatGPT (now claude), my partner as a designer pulls off extremely workable implementations of all different ideas without much need for sketching in a non-functional design tool.

When she does draft in design tools it’s usually more graphic in nature (branding etc).

check out the paper. it's pretty comprehensive https://ai.meta.com/research/publications/the-llama-3-herd-o...

First I would calculate the number of tokens you actually need. If its less than 32k there are plenty of ways to pull this off without RAG. If more (millions), you should understand RAG is an approximation technique and results may not be as high quality. If wayyyy more (billions), you might actually want to finetune

Why not just self sign a local cert? Its really easy to do (ask an AI for a relevant example in your language of choice)

Guh, wish i could delete this now that the title was updated. the original title (shown on the linked page) wasn't super clear

Yep the title was updated.

Kernel is an overloaded term for GPUs. This is about the linux kernel

Apple watch has this feature

I meant 8b -> 8billion rather than 70b

Mine is also an m1. Just use llama3, its 8b quantized by default

I use my mac mini exactly as described by the parent post but using ollama as the server. Super easy setup and obv chatgpt can guide you through it

Best chatgpt could do was deltas by lines https://github.com/bwasti/bram.town/blob/main/server.ts#L142

Perfect is the enemy of dinner

Since macOS High sierra apparently, 2017 release :(

restricts freedom in one of the parameters (A) to make training substantially more efficient (easier for a GPU to churn through).

the actual flops involved are similar to the original SSM-based version, but that's harder to formulate as strictly matrix multiplications