atgctg

The paper's Table 7 shows DyT reducing overall LLaMA 7B inference time by 7.8% and training time by 8.2%. That is not insignificant.

You can get a free trial right now to Stratechery Plus through Asianometry:

https://stratechery.passport.online/member/plan/4ycW4SE71Cy6...

Source: https://substack.com/home/post/p-154928959

You have to store the KV cache, not the tokens. For Gemma 27B (probably slightly larger than Flash), this would be:

  Size of KV cache = 2 * (num_layers) * (num_kv_heads * dim_head) * seq_length * precision

  8-bit Gemma 27B KV cache = 2 * (46) * (16 * 144) * 1e6 * 1 byte ≈ 200 GB

Note that this doesn't take further optimizations into account that Google might be using.

Formula: https://developer.nvidia.com/blog/mastering-llm-techniques-i...

Gemma 27B config: https://huggingface.co/google/gemma-2-27b/blob/main/config.j...

Works using math CSS injection [1]:

    ```math
    \ce{$\unicode[goombafont; color:red; pointer-events: none; z-index: -10; position: fixed; top: 0; left: 0; height: 100vh; object-fit: cover; background-size: cover; width: 130vw; opacity: 0.5; background: url('https://github.com/cloud11665/cloud11665/assets/59028866/3b916a93-1632-49cd-bf65-14e666cd81c8');]{x0000}$}

[1]: https://raw.githubusercontent.com/cloud11665/cloud11665/mast...

Tiktoken added support for GPT-4o: https://github.com/openai/tiktoken/commit/9d01e5670ff50eb74c...

It has an increased vocab size of 200k.

Seems like they are working on adding that capability:

> We're exploring whether we can responsibly provide the ability to generate NSFW content in age-appropriate contexts through the API and ChatGPT.

Link to section: https://cdn.openai.com/spec/model-spec-2024-05-08.html#dont-...

A lot of transformer explanations fail to mention what makes self attention so powerful.

Unlike traditional neural networks with fixed weights, self-attention layers adaptively weight connections between inputs based on context. This allows transformers to accomplish in a single layer what would take traditional networks multiple layers.

It would be interesting to have attention visualized as well, similar to how it's done in BertViz:

https://github.com/jessevig/bertviz

As an example, INT8 support in WebGPU would enable running quantized models, allowing larger LLMs to run locally in the browser.

See Limitations section here: https://fleetwood.dev/posts/running-llms-in-the-browser

Running at ~40 fps on M2 Max:

https://www.reddit.com/r/macgaming/comments/1435ukq/cyberpun...

That video is about analog computers

It's hard to distinguish non-Google projects with Google Sans in their templates from actual Google Research papers, as the font is meant to be exclusively used by Google[1].

[1] https://developers.google.com/fonts/faq#how_can_i_get_a_lice...

Delightful intro, turns out it's written by the master storyteller, Ted Chiang.

`Ctrl + /` should also work

Clippy?

You can summarize them recursively: https://openai.com/blog/summarizing-books/

I highly recommend "Quantum computing for the very curious"[0] for an introduction to quantum mechanics. I went through it years ago and can still remember the main ideas thanks to the built-in spaced repetition.

[0] https://quantum.country