brrrrrm

sure, numerous examples can be shown to say smart play does help. but, would you argue the net benefits of smart play are identical between a sport like basketball and racing?

I wouldn't deny that (and probably should have caveated this in my OP), but compared to a basketball or football team, the benefit of smart play doesn't seem as significant compared to doping up and pressing hard.

> endurance sport which is a natural target for doping

This makes a lot of sense to me. A very singular goal of "maximum output" without much need for fine motor skills and strategizing. I'd guess sprinting/marathons might have similar issues?

emergent behavior. These things are surprisingly good at generalizing

what's going to be interesting is to see the large space of fused kernels being tackled by AI generated code. that might include gemm + relu + gemm + a norm of some kind - which would be annoyingly exhaustive to 1. sweep with a tuner and 2. handwrite as a human

Open-source use of closed source models?

wait sorry, can you explain how this works? I thought gemma3 used siglip, which can output all 256 embeddings in parallel

(also, would you mind sharing a code pointer if you have any handy? I found this https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd... but not sure if that's the codepath taken)

hmm, I'm getting the same results - but I see on M1 with a 7b model we should expect ~10x faster prompt processing

https://github.com/ggml-org/llama.cpp/discussions/4167

I wonder if it's the encoder that isn't optimized?

> When using least squares, a zero derivative always marks a minimum. But that's not true in general ... To tell the difference between a minimum and a maximum, you'd need to look at the second derivative.

It's interesting to continue the analysis into higher dimensions, which have interesting stationary points that require looking at the matrix properties of a specific type of second order derivative (the Hessian) https://en.wikipedia.org/wiki/Saddle_point

In general it's super powerful to convert data problems like linear regression into geometric considerations.

how is this data collected?

a premise*

Do super computers run in fp64 mostly? At fp8 an h100 hits 2 petaflops, and with only 1000 of them you’ve got more compute power than el capitan (in raw flop count)

Paper https://arxiv.org/abs/2501.00663

I probably came in too hot on that (dealing with some personal stuff). Although I disagree with purported the impact of the paper, I don’t think this is fundamentally incorrect or bad science and I wish you the best on future research.

this paper plays into some popular fantasy about the aesthetic of ANNs. it’s not scientifically useful

"y'all" a particularly popular term in NYC (I use it frequently and have lived in NYC most of my life) - so it's kinda bizarre to see "texas monthly" responding to "new york times" as if there is any cultural conflict beyond some random linguist's op-ed.

there was a heatmap published in the past (seemingly only saved by paywalled news organizations) that showed the prevalence of "y'all" in different areas.

probably my biggest pure JS app is this one: https://github.com/bwasti/mebm

you can try it here https://bwasti.github.io/mebm/

your single word firehose demo is awesome. I can see a lot of creative people doing really cool things with this

how are you running CUDA on the integrated Apple silicon GPU these days?

would be fun to do a leaderboard of some specific size (e.g. 4096x4096x4096) just to get all the code and tricks in one spot for folks to learn about things

WebGPU cannot even come close unfortunately since they don't have support for hardware specific memory or warp-level primitives (like TMA or tensorcores). it's not like it gets 80% of perf, it gets < 30% of the peak perf for anything related to heavy compute matrix multiplications

I don't think this is absurd at all, I'm in the exact same boat.

In fact, I suspect most people have far more sophisticated relationships with digital companies these days than ever before. Grievances like cancellation pain are an oversight of antiquated businesses that don't realize it, imo

fake it.

add some latency to the first token and then "stream" at the rate you received tokens even though the entire thing (or some sizable chunk) has been generated. that'll give you the buffer you need to seem fast while also staying safe.

an oracle filtering a random generator will always produce oracle-grade results. is the computer actually getting better or is Lenat just acting as an oracle and the writer is running with it?

that silly softmax1 blog post is not worth the read. no one uses it in practice

if you think about it, the "escape hatch" is the design of the entire transformer dictionary. if Key/Query attention misaligns with Value's weights, you get a layer head that does not attend to anything...

yea. everything. try it out (point it at a website or something)

it's not about the API. its about the documentation + ecosystem.

TF's doesn't seem very good. I just tried to figure out how to learn a linear mapping with TF and went through this:

1. googled "linear layer in tensorflow" and got to the page about linear.

2. spent 5 minutes trying to understand why monotonicity would be a central tenet of the documentation

3. realizing that's not the right "linear" I couldn't think of what the appropriate name would be

4. I know MLPs have them, google "tensorflow mlp example"

5. click the apr '24 page: https://www.tensorflow.org/guide/core/mlp_core

6. read through 10[!] code blocks that are basically just boiler-plate setup of data and visualizations. entirely unrelated to MLPs

7. realize they call it "dense" in tensorflow world

8. see that "dense" needs to be implemented manually

9. think that's strange, google "tensorflow dense layer"

10. find a keras API (https://www.tensorflow.org/api_docs/python/tf/keras/layers/D...)

You almost always have to for good perf on non-trivial operations

depends on your phone, but try a couple of these variants with ollama https://ollama.com/library/llama3.2/tags

e.g. `ollama run llama3.2:1b-instruct-q4_0`

Any citations for that? Id have thought the goofy width of spectacles was related to the screen projection