> "idea atoms"
I made it to the first figure in the paper before I raised an eyebrow at this term.
Then I thought of "Already Clean": https://already-clean.goodfire.com
I cant help but think of Plato's forms -
I made it to the first figure in the paper before I raised an eyebrow at this term.
Then I thought of "Already Clean": https://already-clean.goodfire.com
I cant help but think of Plato's forms -
But yes, we might be bad at marketing. Internally the term we use more often is "concept elements".
The intuition is that if you take a complex treatment of a topic (like an academic paper), you should be able to break it down into some more abstract self-standing ideas. Then, critically, these units are composable and easily embeddable (i.e. they don't have proper names and numbers).
Here's a couple examples of these idea atoms pulled from our paper https://lacuna.tiptreesystems.com/work/the-alien-space-of-sc...: "Scientific ideation often conflates the logical coherence of a research direction with its likelihood of being discovered by the current community. This oversight makes it difficult to identify research paths that are scientifically sound yet unlikely to be discovered due to the existing distribution of expertise and collaboration networks."
Another example: "A generative model estimates the plausibility of concept combinations by treating conceptual units as tokens and calculating their length-normalized log-likelihood. By training on random orderings of these units, the model learns to identify which components are likely to co-occur in a valid research contribution regardless of their sequence."
Goodfire is a discovery from research, an ahaha moment.
"concept elements" would have been a better descriptor in both cases. The V for the cats and the cars, and for your idea.
> 16,068 LLM ... The Coherence Model is a 42.8M-parameter transformer trained
My question is this: was this the smallest model you could train on the 16k papers? Isn't there an optimal small size to identify everything that is common?
Wouldn't it make more sense to use such a model to strip OUT the commonality? Get rid of everything where the "existing distribution of expertise and collaboration networks" are already optimal - and what's left is the novelty?
Ultimately I would be curious to see if your existing "atoms" happen to follow Zipfs law...
I came up with that name, and fair eyebrow. Think of an "atom" as a unit of a working vocabulary (a "token"), not a Platonic form. It's a way of compressing the corpus (paraphrases of the same idea across papers get merged into one symbol, a bit like a tokenizer's vocabulary).
We also talk about some limitations of this framing in page 11 (towards the top).