Deterministic yes, robust deterministic no. The most likely conjunction is not always the best nor representative of what the model is considering unless its certainty is high.
723 karma · joined November 18, 2021
This means when writing documentation, tutorials or commit messages, their output is often a garbled jumble. Assuming shared context, using invented terminology without explaining, leaking conversational states due to improper epistemic boundaries and failing to model the reader. This all usually leads to their freely generated explanations being terrible. Getting good explanations requires chaining questions that force them to line things up properly, which is not easy the less you know. These failures as something LLMs naturally struggle with make sense, given the nature of attention and RL with weak signals from human data.
Math is not merely a collection of proofs, it's a way of understanding. A proof presented in a manner that cannot be incorporated remains useless. It does not make it's way to physics like Riemannian geometry and matrix math did. This is no less true when done by humans too.
Your hallucination conclusion, checking if a proof is one, is exactly the counterproductive cost.
Most of us cannot verify that the claims in the OpenAI lore dump are in fact all correct. It will take tons of work from experts to do this. It took subject expert mathematicians to identify the discrepancy and disconnect in the Navier Stokes proofs, for example. LLMs will struggle to make use of their own proofs or turn them into knowledge that accumulates over time.
The act of proving is often more valuable than the proof itself. Human constraints and limitations force us to invent tools and abstractions that a 100,000 x 1M context swarm can bypass. The tradeoff from that AI swarm advantage is work that doesn't usually lend itself to being built upon. It's like doing all the side quests and reading all the books of an RPG versus min maxing a straight path with a guide. We might try to identify new abstractions, but the fact that we don't get access to CoT and that much of it will be illegible means mining LLM traces for what human mathematicians produce naturally will be a tedious chore.
My gut tells me that a better approach to a calibrated 0-shot classifier than shoehorning DiffusionGemma would be starting from another gemma, T5GemmaV2. Take its encoder and do continual training on an RTD objective and a large relational synthetic data mix. Then finetuning (multi-annotator data will help calibration) on as many proper NLI datasets as possible. That still lacks the DebertaV3 disentangled attention's inductive bias, however.
Jev also has its calibrated predictions component which is important. Temperature scaling is probably the easiest first pass. But there's lots of sensible options to improve on that.
ModernBERT might be the easier, more stable starting point than T5Gemma though.
With the blackhole universe picture, causal boundaries and spacetime's evolution aren't properly accounted for once we step out of a static picture.
Incidentally, the cosmic object we are more likely to be inside of is a white-hole.
†Every paper I've read estimates the average information content of transformer LLMs at about 3-4 bits per parameter. Curiously, biological synapses are also estimated to be about 4-5 bits per synapse, possibly a bit lower.
This guess/explanation predicts that most users won't match what the initial social media hype claims, doesn't discount user experience as simple habituation nor does it assume companies are lying when they say there have been no changes to the model itself (quantization included).
I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.
Personally, I can detect a separation between Sol and Astra (but not as large as that between Opus and Fable). While they can solve most of the same problems, Astra takes less time, is less frustrating to talk to, is cleaner, notices more, spins wheels less and requires less corrections.
Openweight models turned a corner around kimi 2.6, deepseek v4 pro/flash, hy3 and mimo 2.5 pro. Similar to how closed LLMs turned a corner around gpt 5.2 and opus 4.5.
While they remain a step behind closed frontier models, for real world tasks ranging across functional reactive programming, distributed systems, mathematical modeling, to-the-millisecond highly optimized spatial data-structures, complex compute shaders and shader effects and non-trivial systems involving parser combinators and algebraic effect systems, I can say that open models have very recently gone from useless to productive. For my work, mimo v2.5 pro is hands down better than sonnet 4.6.
> Men will set the goals and supply the motivations, of course, at least in the early years. They will formulate hypotheses. They will ask questions. They will think of mechanisms, procedures, and models. They will remember that such-and-such a person did some possibly relevant work on a topic of interest back in 1947, or at any rate shortly after World War II, and they will have an idea in what journals it might have been published. In general, they will make approximate and fallible, but leading, contributions, and they will define criteria and serve as evaluators, judging the contributions of the equipment and guiding the general line of thought.
> In addition, men will handle the very-low-probability situations when such situations do actually arise. (In current man-machine systems, that is one of the human operator's most important functions. The sum of the probabilities of very-low-probability alternatives is often much too large to neglect. ) Men will fill in the gaps, either in the problem solution or in the computer program, when the computer has no mode or routine that is applicable in a particular circumstance.
> The information-processing equipment, for its part, will convert hypotheses into testable models and then test the models against data (which the human operator may designate roughly and identify as relevant when the computer presents them for his approval). The equipment will answer questions. It will simulate the mechanisms and models, carry out the procedures, and display the results to the operator. It will transform data, plot graphs ("cutting the cake" in whatever way the human operator specifies, or in several alternative ways if the human operator is not sure what he wants). The equipment will interpolate, extrapolate, and transform. It will convert static equations or logical statements into dynamic models so the human operator can examine their behavior. In general, it will carry out the routinizable, clerical operations that fill the intervals between decisions.
https://www.organism.earth/library/document/man-computer-sym...
let v0 = 0
let v1 = 0.40978399*(0.616*u + 0.291*v)
let v2 = if 0 > v1 then 0 else v1
let v3 = 0
let v4 = 0.377928*(0.261*u + 0.468*v)
let v5 = if 0 > v4 then 0 else v4...It's a subtle difference from what you said in that it's not like everything has to go right in a sequence for the defensive side, defenders just have to hope they committed enough into searching such that the offensive side has a significantly lowered chance of finding solutions they did not. Both the attackers and defenders are attacking a target program and sampling the same distribution for attacks, it's just that the defender is also iterating on patching any found exploits until their budget is exhausted.
Diffusion also won't help the way you seem to think it will (that the outputs occur in a sequence is not relevant, what's relevant is the underlying computation class backing each token output, and there, diffusion as typically done does not improve on things. The argument is subtle but the key is that output dimension and iterations in diffusion do not scale arbitrarily large as a result of problem complexity).
Compression artifacts (which are deterministic distortions in reconstruction) are not the same as hallucinations (plausible samples from a generative model; even when greedy, this is still sampling from the conditional distribution). A better identification is with super-resolution. If we use a generative model, the result will be clearer than a normal blotchy resize but a lot of details about the image will have changed as the model provides its best guesses at what the missing information could have been. LLMs aren't meant to reconstruct a source even though we can attempt to sample their distribution for snippets that are reasonable facsimiles from the original data.
An LLM provides a way to compute the probability of given strings. Once paired with entropy coding, on-line learning on the target data allows us to arrive at the correct MDL based lossless compression view of LLMs.
In the case you're working as part of team large enough to have dedicated programmers, the majority of the roles will usually be in content creation, design and QA.
Think of the negative reputation the Unity engine gained among gamers, even though a lot of excellent games and even performant games (DSP) have been made with it.
More competitors does also raise the bar required for novelty, so it is possible that standards are also rising in parallel.
In my experience, the issue of sycophancy has been longest in the Anthropic models, so it might be most deeply rooted for them. It's only recently, perhaps with the introduction of user A/B preference tests such as by lmarena and the providers themselves has this become a major issue for most other LLMs.
Thinking that simple actions like adding an anti-evil vector to the residual stream to improve behavior sounds naively dangerous. It would not surprise me if unexpected and unwanted downstream effects resulted from this; which a future paper will address too. Not unlike what happened with tuning for user preference.
> V3/R1 scale models as a baseline, one can produce 720,000 tokens
On what hardware? At how many tokens per second? But most importantly, at what quality? I can use a PRNG to generate 7 billion tokens at a fraction of the energy use of an LLM but those tokens are not going to be particularly interesting. Simply counting how many tokens can be generated in a given time frame is still not a like for like comparison. To be complete, the cost required to match human level quality, if possible, also needs accounting for.
> Deeply thinking humans expend up to a a third of their total energy on the brain
Where did you get this from? A 70B LLM? It's wrong or at best, does not make sense. The brain barely spends any more energy above its baseline when thinking hard (often not much more than 5%). This is because most of its energy use is spent on things like up-keep and maintaining resting membrane potential. Ongoing "Background activity" like the DMN also means the brain is always actively computing something interesting.
Also, JL is only a part of the story for the transformers.
I find Schmidhuber's claim on GANs to be tenuous at best, but his claim to have anticipated modern LLMs is very strong, especially if we are going to be awarding nobel prizes for Boltzmann Machines. In https://people.idsia.ch/%7Ejuergen/FKI-147-91ocr.pdf, he really does concretely describe a model that unambiguously anticipated modern attention (technically, either an early form of hypernetworks or a more general form of linear attention, depending on which of its proposed update rules you use).
I also strongly disagree with the idea that his inability to practically apply his ideas held anything back. In the first place, it is uncommon for a discoverer or inventor to immediately grasp all the implications of and applications of their work. Secondly, the key limiter was parallel processing power; it's not a coincidence ANNs took off around the same time GPUs were transitioning away from fixed function pipelines (and Schmidhuber's lab were pioneers there too).
In the interim, when most derided Neural networks, his lab was one of the few that kept research on Neural networks and their application to sequence learning going. Without their contributions, I'm confident Transformers would have happened later.
> It's clear to me no one read his early paper's when developing GANs
This is likely true.
> self-supervision/transformers.
This is not true. Transformers came after lots of research on sequence learners, meta-learning, generalizing RNNs and adaptive alignment. For example, Alex Graves' work on sequence transduction with RNNs eventually led to the direct precursor of modern attention. Graves' work was itself influenced by work with and by Schmidhuber.
It's possible that someone already on the verge of a break or otherwise in a fragile state of mind asking for help with their theories could end up with an LLM telling them how incredibly groundbreaking their insights are, perhaps pushing them quicker, deeper more unmoored in the direction they were already headed.
Between the non-uniformity of receptor density (eg fovea vs peripheral vision but this is general across all senses), dynamic receptor fields and the fact that information is encoded in terms of spike rate and timing patterns across neural populations, the idea of pixels in some bitmap at some resolution is beyond misleading. There is no pixel data, just sparsely coded feature representations capturing things like edges, textures, motion, color contrast and the like, already, at the retina.
While hundreds of trillions of photons might hit our photoreceptors, > 99% of that is filtered and or compressed before even reaching retinal ganglion cells. Only a tiny fraction, about 10 million bits/sec, of the original photon signal rate is transferred through the optic nerve (per eye). This pattern of filtering and attentive prioritization of information in signals continues as we go from sensory fields to thalamus to higher cortical areas.
So while we might encounter factoids like: on the order of a billion bits per second of data hit photoreceptors or [10Mb/s transferred](https://www.britannica.com/science/information-theory/Physio...) along optic nerves, it's important to keep in mind that a lot of the intuition gained from digital information processing does not transfer in any meaningful sense to the brain.
In the top 10, are models by OpenAI (gpt4omini), Google (gemini flashes and pros), Anthropic (Sonnets) and Deepseeks'. Even though the company list grows shorter if we instead look at top model usage grouped by order of magnitude, it retains the same companies.
Personally, the models meeting my quality bar are: gpt 4.1, o4-mini, o3, gpt2.5pro, gemini2.5flash (not 2.0), claude sonnet, deepseek and deepseek r1 (both versions). Claude Sonnet 3.5 was the first time I found LLMs to be useful for programming work. This is not to say there are no good models by others (such as Alibaba, Meta, Mistral, Cohere, THUDM, LG, perhaps Microsoft), particularly in compute constrained scenarios, just that only Deepseek reaches the Quality tier of the big 3.
Language: Syntax errors rise, and a common form is the syntax of a more common language bleeding through.
Domain: Less so than what humans deem complex, quality is more strongly controlled by how much code and documentation there is for a domain. Interesting is that if in a less common subdomain, it will often revert to a more common approach (for example working on shaders for a game that takes place in a cylinder geometry requires a lot more hand-holding than on a plane). It's usually not that they can't do it, but that they require much more involved prompting to get the context appropriately set up and then managing drifting to default, more common patterns. Related is decisions with long term consequences. LLMs are pretty weak at this. In humans this one comes with experience, so it's rare and an instance of low coverage.
Dates: Related is reverting to obsolete API patterns.
Complexity: While not as dominant as domain coverage, complexity does play a role. With likelihood of error rising with complexity.
This means if you're at the intersection of multiple of these (such as a low coverage problem in a functional language), agent mode will likely be too much of a waste for you. But interactive mode can still be highly productive.