My point is exactly that we cannot even begin to start the process that will lead to “know everything” there is to know unless we make it a science first. Ad hoc statistical models are different than understanding or complete knowledge of a subject.
The early physics background is messy and incorrect. I didnt read the full position paper, but from its start: The Lorentz transformations were by Lorentz, well before Einstein’s paper on special relativity; the principle of relativity also existed before the Einstein paper. The math was all there, with steps taken by Maxwell, Voigt, Larmor, Lorentz, and Poincare. Einstein supplied a clean physical interpretation, making all inertial frames equivalent, making simultaneity frame dependent, and explaining length and time deformations without the need of the concept of ether. Skimming the end of the paper with the arguments about lack of abduction or inability to make the analogy without sensory experience, I see that this paper is unfounded speculation rather than solid/hard philosophical logic. As a position paper it is OK to appear, but i think it misses the point of how LLMs or other autoregressive learners of future states can build analogies and intuition that can help them formulate new theories of the world. Soon it will be more obvious to everyone, so I am not very worried about these writings.
Agreed in principle for today’s new ones, though with all the buying frenzy it quickly becomes hard to procure anything but the latest hardware and the economics of older generation hardware makes little sense for new data center deployments.
I agree these are central components, but to avoid oversimplification and the mistaken belief that modern LLMs do a lot of search during inference: If it was so simple, the traditional computer algebra systems would have reached similar breakthroughs when deployed at large supercomputer centers. This didnt happen because the search space is huge. You definitely also need a fancy learning algorithm. Although these ingredients would suffice (depending on what the learning algorithm is), you probably also want to learn in the absense of a strong verifier at every step, to allow building a fuzzy/erratic sense of the search space that can lead to planning/intuition and allow distant jumps in a targetted direction.
I wish we knew everything about how LLMs work! We only know very basic elements related to their construction and traning dynamics, and pretty much every major question we would like to address still has unknown or vague heuristic answers. This is expected for such a young field of study. In physics, we know the Schrodinger equation, but we dont know everything about how the world works or how to create new materials even though we know that these materials are composed by atoms and we can simulate small collections of them. In cell biology, we know the sequences that make up the DNA of a cell and we approach the time we can build minimal synthetic cells with pieces we understand, but we only scratch the surface of our level of understanding of how the cells actually work and new discoveries are added every day. In biology at large we still keep finding new types of tubes inside human brains—not sure what you mean by plenty, but we certainly have an extremely limited understanding of human cognition compared to what we might have in 50 years from now. It is not just anout LLMs and intelligence—I would like us to be able to answer practical questions about how LLMs work in order to improve general or specialized LLMs even faster than today. We “know” about scaling in an empirical sense, and it certainly has a long way to go, but it does not feel close to a complete understanding.
I agree boring problems exist; bounds may have a fare share of them. None of the bounds problems in this set are even close to this category; many of them are closer to the type of contributions that in the past got recognized by special awards. Your initial replies were misleading.
Use nvidia hardware instead and use a larger cluster serving many more users concurrently. Easily 10x–20x higher token rate per GPU with public solutions like dynamo and sglang.
Your inverted logic does not hold. The fact that such useless problems for bounds exist does not mean that improving bounds is useless. 9 fields medals in the last twenty years, including the one to Terrence Tao, were for improvements on bounds. 3 of the 4 medals in 2022 were for bounds; 2 of these medals were in combinatorics.
It is unfair to dismiss contributions to decades old open problems as equivalent to calculating more digits of pi. It missed the mark by a lot—as does the two bucket simplifaction.
> Whilst current models can't 'intuit' and come up with conjectures
I disagree. I routinely let LLMs speculate or generate hypotheses along the way of helping with technical research. Sometimes they can prove the correctness of a concrete math idea but other times even an unproven conjecture helps with the numerical algorithm implementation and the result is then simply supported by additional data. I guess that any autoresearch-adjacent application has LLMs intuiting and coming up with hypotheses/conjectures—as do the steps/lemmas along a complex proof. In my opinion the modern LLMs are powerful intuitive thinkers that generate lots of conjectures of varying quality or importance.
Not sure what your first sentence means, or why you are quoting AI. Many of these problems individually were math at a level approaching the highest possible for expert human mathematicians. These are not simple combinations of existing ideas, or following of human intuitions, or implementing something following specific human instructions. Then again maybe you mean that math is not knowledge and that all math is simply extending the basic axioms using known patterns, to which I would not agree.
It is much more subtle than your specific example, which is strictly speaking a bug, though ofc it has been used during pretraining for efficiency purposes. Sglang and miles have been working towards full determinism in open source codebases, so the LLMs can help explain the subtleties encountered in actual projects if you point them to these repos.
In the simplest possible case, a distributed addition of floating point numbers is not deterministic if you don't specify the order of the addition operations. If floats are added in a first-come first-serve fashion (the simplest implementation of a reduction operation) you already lose determinism. These could be activations from multiple experts, but it could also be adding chunks of a matrix operation that uses multiple CUDA cores.
If you manage to achieve determinism in temperature zero, it is possible to extend it to determinism at higher temperatures, because at that point you only have to keep the pseudo-random-number generator state in sync across parallel instances and this problem has been solved.
You can do whatever you want with the model within your own organization. If you use it commercially—either as a model-as-a-service business or in a very large-scale product—you should check the additional license terms, which go beyond MIT. My interpretation is that Moonshot cares about the exact inference behavior and accurate representation of their model or derivatives, and perhaps also about capturing some additional value despite their own GPU limitations, so the extra license terms focus on those large-scale commercial deployments.
Or just fix it. I use lockdown mode on iphone. It does not work on safari or chrome, ironically telling me to use a different browser like safari or chrome.
Serious question: if one were willing to give up on curses, isn’t Emacs/elisp providing the best multiplexing system available to humankind? And conceptually, why would agents ever need to depend on curses?
The drug candidates that enter human clinical trials, (phase 1 to phase 3) are identical in all ways to the final product if approved. The company is not allowed to change any part of the manufacturing process. All the research for how to make the drug is done in preclinical phases. In the clinical phase we test the unknown human biology with a final product candidate.
The internet convention in the 90s used to be two dashes for the en-dash character and three dashes for em-dash character. You used a space-dash-space notation that is easy to decipher but the typical dashes variants dont have spaces around them. When I read a text message I dont expect careful typography—in a book or an article I want perfection.
More importantly though, the em-dash-in-a-sentence construct is a way of thinking and the notation is of lesser importance than the way to express oneself. I dont think the current distrust is from the symbol: the LLMs love em dashes for the same reasons many of us do, and, perhaps better than most of us, they have no trouble using the correct symbol.
It was never DeepSeek’s release that dropped the NASDAQ at the time; it was the unknown risks of the early thoughts related to trade wars. Popular financial newspapers can promote anything they want, but these news do not typically drive large investor decisions.
The blog is about writing an agent when you dont already have an agent, but only a plain LLM. It stitches the minimal pieces together. Agents dont need lots of supporting infra, so it is good to keep the code concise. Not a wow moment for sure, though some people think that agents and harnesses are complicated.
If you look at the timelines for the hiring of the hardware team, this was an extremely fast and high risk implementation from concept to tapeout. Amazing it works at all during bringup.
Regarding lack of p-hacking. The placebo arm of blinded trials breaks when your brain can detect a medication. The effect is tiny in these studies; approval was rushed to give hope to patients. The drug was discontinued later.
I work in drug discovery. For the past twenty years or so, my personal analogy for this hypothesis has been a fantasy story around the days after the bombing of Dresden, when a new civilization suddenly visits Dresden and has no priors about what may have happened there. The aliens see bricks all over the place and assume that the bricks were the cause of the catastrophe. They take great efforts to pick up the bricks and save a couple of lives from the people who were covered in the ruble. The aliens build better systems to pick up bricks in the future and get ready to act next time. When a nearby city gets bombed, they quickly visit and help recover bricks saving a couple more lives. A different civilization could have instead focused on reducing the bombs or detecting and defending against the attacking airplanes.
Our immune systems are complicated, much more so than airplanes and bombs. The amyloid deposits are very likely part of an immune response, and although in principle immune responses going wild are horrible and can be fixed, it is very important to work on identifying and addressing the causal factors of this disease. There have been more therapies tested on the amyloid hypothesis that mere statistical fluctuations could explain away. I don't always agree with Derek, but I'm with him on this one. New ideas are urgently needed here, or this horrible disease will be an increasingly common end state for our aging populations.
Non-US models are not banned in the US: they are used daily in every state of the US. Some misguided state governments temporarily banned employees from downloading the R1 models and variants released 16 months ago on state government computers. The article and your comment are misleading :-)
The GP did not try to time the market. He suggested a sensible strategy to exclude a tiny subset from an index (less expensive than maintaing the alternative index yourself).
Frankly, everyone in the industry knows. When people make these statements without additional clarity they always talk about API prices. You can look at the NVL72 specs and make estimates for electricity and ownership costs rather easily. Inference at data-center scale is dirt cheap, even with public codes using dynamo and sglang. The mystery is why the early misconceptions about inefficient inference persisted even after NVIDIA was very open about everything they did to help reduce costs dramatically in the last two years.