935 karma · joined March 6, 2023
Another takeaway is that you don’t need to pass a tensor of shape (batch_size, sequence_length, d_model) through your transformer. Not every token needs its own dedicated latent embedding. You can presumably get away with dividing sequence_length by a constant.
This isn’t super ground breaking but it does reinforce the validity of a middle ground between recurrent models, where context is compressed into a single “memory token”, and transformers, where context is uncompressed. 1 < n/k < n
But under the hood, the image will have to be transformed into features / embeddings before it can be decoded into text. Suppose that the image gets processed into 100 “image tokens”, which are subsequently decoded into 1000 “text tokens”.
Now forget that we are even talking about images or OCR. If you look at just the decoding process, you find that we were able to compress the output into a 10x smaller representation.
The implication for LLMs is that we don’t need 1000 tokens and 1000 token embeddings to produce the 1001st token, if we can figure out how to compress them into a 10x smaller latent representation first.
Discrete hierarchical representations are super cool. The pattern of activations across layers amounts to a “parse tree” for each input. You have effectively compressed the image into a short sequence of integers.
Judging by how often Sam Altman makes appearances in DC, it's not just money that sets OpenAI apart. It's likely also a strategically important research and development vehicle with implicit state backing, like Intel or Boeing or Palantir or SpaceX. The losses don't matter, they can be covered by a keystroke at the Fed if necessary.
In the meantime I would like to see more Hoover Dams and Apollo programs here, with fewer tent cities and high-speed rail boondoggles. This will require a ruling class that 1) is competent and 2) cares for the people it governs.
I think of DNA analogously to the rules of cellular automata. The entropy of the rules is much less than the entropy of the dynamical system the rules describe.
The body is filled with innate knowledge. The organs all know what to do. The immune system learns to detect intruders (without synapses). Even a single cell organism is capable of complex and fluid goal-oriented behavior, as Michael Levine attests.
I think the assumption that all knowledge exists in the brain, and all knowledge in the brain is encoded by neuronal weights, is probably too simplistic.
Regarding language and vision, I think the cognitive scientists are right: it is better to view these as organs or “modules” suited to a function. Damage Broca’s area and you get Broca’s aphasia. Damage your lung and you get trouble breathing. Neither of these looks like the result of statistical learning from randomly initialized parameters.
Also I’m skeptical that self-supervised learning is sufficient for human level learning. Some of our ability is innate. I don’t believe it’s possible for statistical methods to learn language from raw audiovisual data the way children can.
For graph/tree document representations, it’s common in RAG to use summaries and aggregation. For example, the search yields a match on a chunk, but you want to include context from adjacent chunks — either laterally, in the same document section, or vertically, going up a level to include the title and summary of the parent node. How you integrate and aggregate the surrounding context is up to you. Different RAG systems handle it differently, each with its own trade offs. The point is that the system is static and hardcoded.
The agentic approach is: instead of trying to synthesize and rank/re-rank your search results into a single deliverable, why not leave that to the LLM, which can dynamically traverse your data. For a document tree, I would try exposing the tree structure to the LLM. Return the result with pointers to relevant neighbor nodes, each with a short description. Then the LLM can decide, based on what it finds, to run a new search or explore local nodes.
The West is run by finance capital, which employs the lawyers (and buys the politicians).
There was a post here a while back about engineering grads in the UK who couldn't get engineering jobs. So they ended up working for quant firms and banks instead.
Under neoliberalism the economy ends up oriented away from productive activity and toward rent-seeking and wealth transfers. Hence the growing gambling "industry", the pump and dump crypto scams (run by heads of state, no less), the legally protected private cartels like banking and medicine. We get people like Vivek Ramaswamy who became a billionaire while producing nothing of value.
Pinning these massive systemic issues on lawyers is frankly stupid. They are just one piece of the puzzle.
This is basically the thesis of Bertrand de Jouvenal's "On Power" (1945).
It's hard to see how that can happen when politicians take money from the rent-seekers who benefit from the status quo. "Competition is for losers", says Peter Thiel, so buy yourself a state-sanctioned monopoly (like Palantir).
It seems plausible and intuitive that simple tools dynamically called by agents would yield better results than complex search pipelines. But do you have any hard data backing this up?
The Ancient Greek model of politics isn't compatible with liberal pluralism. The former assumes a common end and the latter assumes diverse conflicting ends. The Ancient Greek model looks more like modern China than it does like modern America or Europe.
Maybe the arrows are a video game reference? I still don’t see a groyper tie there.
Meanwhile mainstream sources are reporting that the shooter appeared to be motivated by left political ideology.
https://www.theguardian.com/us-news/live/2025/sep/12/charlie...
https://en.wikipedia.org/wiki/Three_Arrows
The groypers are not antifascist, so the reported symbolism doesn't add up. If there's evidence of the shooter being a groyper, then I'm curious to see it.