225 karma · joined July 7, 2013
Email is gmail.
As for how to get more adoption of property-based testing and/or stateful and/or parallel property-based testing, is there any way to do this from outside the language level? So you'd write your tests to consume commands and produce results from stdio, then an external process can drive the testing process. Can you run these in a VM to get deterministic thread scheduling for any language? Fakes could be implemented in any language and use the same stdio mechanism. And then build up a public set of known-good fakes. Ideally the fakes could be composable but not sure that is easy. Of course for bespoke things you'll need a bespoke fake, but I am thinking of the general stuff like a dictionary, queue, etc. This is all sparked because I've been thinking of how to build a giant inter-language inter-library testing database, initially with unit tests but this made me think about how to incorporate property based tests.
3.1 Optimised attention: Instead of using a learned W_V to project from C to H, slice V into H sized vectors. (V is just the input tokens X). This is because the matrix multiply is to a lower dimension anyway, so why not just slice. Slicing is just reshaping (L, C) -> (L, nh, H)
3.2 Efficient attention: I think this opens with a typo, "In the last section, we discussed how and why we can remove W_O..." should be W_V not W_O I think. Anyways, same as above, just for the keys this time. Reshape K (which is just X) from (L, C) -> (L, nh, H)
3.3 Super attention: Introduce an (L, L) W_A (lower triangular for masked) that transforms V on the left (X again) from (L, C) -> (L, C) (whereas standard attention has W_V (C, C) that transforms (L, C) -> (L, C) from the right). And they share W_A between heads.More efficient when C > L, so for long context models, probably not more efficient.
I think the first two modifications are equivalent to just setting W_V and W_K to the constant identity matrices right? So that makes me think what would happen if you instead restrict W_V (and/or W_K, W_Q) to be block diagonal (though non square) such that each head has in effect an (H, H) matrix which transforms the slice of X it receives. This is different than standard attention right? Because there the W_V acts over the full C dimension. Almost surely someone has thought of this so I will try to find out
Still learning so all this could be wrong
> We call it a superconductor processing unit (SPU), with embedded superconducting SRAM, DRAM memory stacks, and switches, all interconnected on silicon interposer ... Next, there is a glass bridge to a warmer area, a balmy 77 K that hosts the DRAM.
Having 20 exaflops in a 20x20x12 cm volume is cool, but aren't you going to need a memory bandwidth close (factor 10 - 1000 lets say depending on arithmetic intensity) to that to be useful? And total memory capacity as well. I feel like the bytes/second/area (bandwidth flux) would be the limiting factor to make use of that compute density
From the linked paper https://pubs.aip.org/aip/apl/article-abstract/122/18/182604/... the projected JSRAM density is 4MB/cm2 so barely anything on the SRAM front
Is this accurate? I thought for example gemini pro used image tokens and gpt4-o similar
> without the need for separate image/text encoders
but then they say they pre-trained two different tokenizers, so maybe they just mean that the tokens go into the same attention layer? But then I thought that is how all the multi-modal stuff was happening already?
two typos stabilitize and multiplicate
They make a big deal about these auxiliary constructions but I'm not sure the results back that up. I also don't understand what the "without pretraining" means; they say without pretraining it gets 21 problems, without fine-tuning 23 problems, and altogether 25. So are they getting 21/25 without their synthetic data at all??
I want to look at how the dependency graph is encoded in the training examples and whether you might expect different results from different encodings. Ideally I'd think you'd want that part of the model/embedding to be invariant across all possible (big number!) of encodings.
The human comparisons are really weak in Fig 2 and unfortunately is probably what the media mainly focuses on. I do find the results interesting and wonder how well this translates into program synthesis for example. The paper mentioned in these comments which I have yet to read more "Language Models Can Teach Themselves to Program Better" sounds related. I'd be very interested to see more methods that are very different from LLM heuristic beam search, even though it does seem compelling and they get to publish lots of papers with it.
To me, Trumpet sounds more like a compiler or circuit/program synthesizer than an operating system, since my understanding is you have some circuit in mind and the goal is to find a good genome (and/or the enzymes, not super sure yet) that can implement it. This means finding enzymes that will accurately promote or block transcription at a target location. And also checking against potential problems as the computation/reaction progresses since in a deep circuit depth you would presumably have many more kinds of things floating around that could interfere. I didn't see any mention on the circuit efficiency here, like how many base pairs per gate as # of gates increases, which I would guess would get larger faster than linear.
They mention the signal amplification of many genomes being transcribed which is really neat (like error correction). Though makes me wonder if the output signal has a defined or expected duration of stability. If it is stable forever, it is like a fixed point or attractor, but I suspect unless under much greater design or control, things would change. And what is the latency of these gates?
With cell-free, it seems plausible to design from-scratch genomes using a database of known proteins and that could be really cool. And funny to think about applying evolutionary search to determine the fittest genome for the job. From there we go to unicellular and then up to multicellular!
This is only about predicting performance of a given graph and not about improving/suggesting/editing a new equivalent graph. As in FunSearch, models which have decent predictive power could be used with evolutionary search.
[1] https://github.com/google-research-datasets/tpu_graphs#featu...
[2] TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs https://arxiv.org/abs/2308.13490
[3] Learning Large Graph Property Prediction via Graph Segment Training https://arxiv.org/abs/2305.12322