you missed the boat on natural language being the new interface
best orgs will own their data and have full history in version control so that it's easier for LLMs and humans to work with, not walled garden traps
4,703 karma · joined February 20, 2012
best orgs will own their data and have full history in version control so that it's easier for LLMs and humans to work with, not walled garden traps
most open claw users have no idea how easy it is to add backdoors to these models and now they're getting free reign on your computer to do anything they want.
the risks were minimal with last generation of chat models, but now that they do tool calling and long horizon execution with little to no supervision it's going to become a real problem
> They won't, but this is the actual reason. Nobody likes dealing with support or maintenance, and having to reach out to tiny teams is death by a million papercuts for the end user too.
you will have thousands of linear like products eating the slow moving jiras of the world. great small product driven teams, not slop thrown together by your mom
AI raises the ceiling much further than the floor and it raises the floor a ton. the best software, movies, etc will still be produced by experts in their field, they'll just be able to do way more for less.
the bottleneck at large orgs is communication already, this will get even worse when time to produce stuff goes way down. big cos will drown in slop and are probably better off starting from scratch
it's the end of 80-90% margins that the valley coasted on for the last 20 years. Salesforces of the world will not lose to an LLM, they will lose to thousands of tiny teams that outship them and beat them on cost
instead of 7 figure contracts you'll have customized tailored tools for enterprises, and on the other end you'll have a custom nearly free CRM for every persona
this also means that VCs will stop investing in it, unless it's a platform with network effects and heavy lock in
Some recent works that people might find interesting:
- Evolution Strategies at the Hyperscale - https://eshyperscale.github.io/
- Introducing Nested Learning: A new ML paradigm for continual learning - https://research.google/blog/introducing-nested-learning-a-n...
- Less is More: Recursive Reasoning with Tiny Networks - https://arxiv.org/abs/2510.04871
- Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs - https://arxiv.org/abs/2511.16664
Current model architectures are optimized to get the most out of GPUs, which is why we have transformers dominating as they're mostly large dense matrix multiplies.
There's plenty of work showing transformers improve with inner dimension size but it's not feasible to scale them up further because it blows up parameter and activation sizes (including KV caches) so people to turn to low rank ("sparse") decompositions like MLA.
Lottery ticket hypothesis shows that most of the weights in current models are redundant and that we could get away with much smaller sparse models, but currently there's no advantage to doing so because on GPUs you still end up doing dense multiplies.
Plenty of mech interp work shows that models are forced to commingle different concepts to fit them into the "low" dimensional vector space. (https://www.neelnanda.io/mechanistic-interpretability/glossa...)
Brains do it somehow, so sparsely / locally activated architectures are probably the way to go long term, but we're decades away from that being commercially viable.
https://arxiv.org/abs/1803.03635
EDIT: don't have time to write it up, but here's gemini 3 with a short explanation:
To simulate the brain's efficiency using Transformer-like architectures, we would need to fundamentally alter three layers of the stack: the *mathematical representation* (moving to high dimensions), the *computational model* (moving to sparsity), and the *physical hardware* (moving to neuromorphic chips).
Here is how we could simulate a "Brain-Like Transformer" by combining High-Dimensional Computing (HDC) with Spiking Neural Networks (SNNs).
### 1\. The Representation: Hyperdimensional Computing (HDC)
Current Transformers use "dense" embeddings—e.g., a vector of 4,096 floating-point numbers (like `[0.1, -0.5, 0.03, ...]`). Every number matters. To mimic the brain, we would switch to *Hyperdimensional Vectors* (e.g., 10,000+ dimensions), but make them *binary and sparse*.
* **Holographic Representation:** In HDC, concepts (like "cat") are stored as massive randomized vectors of 1s and 0s. Information is distributed "holographically" across the entire vector. You can cut the vector in half, and it still retains the information (just noisier), similar to how brain lesions don't always destroy specific memories.
* **Math without Multiplication:** In this high-dimensional binary space, you don't need expensive floating-point matrix multiplication. You can use simple bitwise operations:
* **Binding (Association):** XOR operations (`A ⊕ B`).
* **Bundling (Superposition):** Majority rule (voting).
* **Permutation:** Bit shifting.
* **Simulation Benefit:** This allows a Transformer to manipulate massive "context windows" using extremely cheap binary logic gates instead of energy-hungry floating-point multipliers.
Standard Attention is $O(N^2)$ because it forces every token to query every other token. A "Spiking Transformer" simulates the brain's "event-driven" nature.
* **Dynamic Sparsity:** Instead of a dense matrix multiplication, neurons would only "fire" (send a signal) if their activation crosses a threshold. If a token's relevance score is low, it sends *zero* spikes. The hardware performs *no* work for that connection.
* **The "Winner-Take-All" Circuit:** In the brain, inhibitory neurons suppress weak signals so only the strongest "win." A simulated Sparse Transformer would replace the Softmax function (which technically keeps all values non-zero) with a **k-Winner-Take-All** function.
* *Result:* The attention matrix becomes 99% empty (sparse). The system only processes the top 1% of relevant connections, similar to how you ignore the feeling of your socks until you think about them.
Even if you write sparse code, a standard GPU (NVIDIA H100) is bad at running it. GPUs like dense, predictable blocks of numbers. To simulate the brain efficiently, we need *Neuromorphic Hardware* (like Intel Loihi or IBM NorthPole).
* **Address Event Representation (AER):** Instead of a "clock" ticking every nanosecond forcing all neurons to update, the hardware is asynchronous. It sits idle (consuming nanowatts) until a "spike" packet arrives at a specific address.
* **Processing-in-Memory (PIM):** To handle the high dimensionality (e.g., 100,000-dimensional vectors), the hardware moves the logic gates *inside* the RAM arrays. This eliminates the energy cost of moving those massive vectors back and forth.
| Feature | Standard Transformer | Simulated "Brain-Like" Transformer | | :--- | :--- | :--- | | *Dimension* | Low (\~4k), Dense, Float32 | *Ultra-High* (\~100k), Sparse, Binary | | *Operation* | Matrix Multiplication (MACs) | *Bitwise XOR / Popcount* | | *Attention* | Global Softmax ($N^2$) | *Spiking k-Winner-Take-All* (Linear) | | *Activation* | Continuous (RELU/GELU) | *Discrete Spikes* (Fire-or-Silence) | | *Hardware* | GPU (Synchronous) | *Neuromorphic* (Asynchronous) |
The whole US economy is getting propped up by AI spending, and to continue on the scaling law ladder each new iteration requires 10x more investment. If OpenAI is not able to raise it's over for anyone adjacent to them (NVIDIA, AMD, Anthropic, etc.).
It's AGI or bust for OpenAI, because as you said there's no margin competing on API tokens since open source is almost as good for most use cases.
Keep raising 10x more for each round of scaling and very quickly you get large enough to be able to bully anyone into playing with you.
Sama got big enough to be able to twist any arm he wants and soon OpenAI will be too large to fail.
This bubble popping will definitely take down crypto with it and rip through other adjacent industries.
My point was that the large players have monopoly hold on large swaths of the internet and are using it to further advantage themselves over the competition. See Veo 3 as an example, YouTube creators didn’t upload their work to help Google train a model to compete with them but Google did it anyways, and creators didn’t have a choice because all eye balls are on YouTube.
It was for a few specific ML research roles that I was interested in, of which there were very few in NYC and during the interview process I was told that they would go to internal candidates
Just like the dot com bubble we'll need to wash out a ton of "unicorn" companies selling $1s for $0.50 before we see the long term gains.
Pretty much give away Sonnet level coding model and have it work with GPT-5 for harder tasks / planning.