Hacker News Reader

steppion Foundation Model Engineering: From Theory to Production

The GitHub README has this disclaimer:

Translation Notice: The Korean version of this resource has been translated with the assistance of Gemini. As a result, some phrasing might feel unnatural. We actively welcome improvements and corrections! Please refer to our CONTRIBUTING.md guide to see how you can contribute to making this resource better.

Assuming this means the Korean is the original, I'm curious how much effort was put into the Korean text and whether it reads naturally and maintains thematic unity. A serious book on this subject matter would fill a valuable need right now

steppion Principia Mathematica is modern and insightful

A failed hypothesis is not the same thing as a logical error. Would they have been more free of flaws if they believed it was impossible and thus didn't try? We'd all be the poorer for it

steppion Principia Mathematica is modern and insightful

Wittgenstein didn't find logical flaws in the Principia and deeply admired it. He found flaws in Russell's follow up work on Epistemology, "The Theory of Knowledge."

steppion Principia Mathematica is modern and insightful

This is commonly believed, but Gödel didn't identify a logical error at the heart of the whole enterprise, he proved astonishing theorems revealing limitations of any sufficiently powerful formal system. One can kind of think of the Principia as a science experiment to find the extent to which known mathematics could be proven from foundational axioms that could be thought of as "laws of logic". To make their system work, Russell and Whitehead themselves had to add extralogical axioms, such as their Axiom of Reducibility [0] and the Axiom of Infinity, giving empirical evidence (but not a proof) that "laws of logic" alone were not enough. They were also aware of limitations in their own system, such as the inability to define the cardinal $\aleph_\omega$ [1].

Like the article says, what they did was ahead-of-its-time, and a monumental influence on all subsequent work on formal systems, including Gödel's work, regardless of whether Russell and Whitehead achieved their initial aims.

[0] https://en.wikipedia.org/wiki/Axiom_of_reducibility [1] https://www.gutenberg.org/files/78255/78255-h/78255-h.htm#Pa...

steppion Mathematics Distillation Challenge – Equational Theories

Tao does state his hopes in the article: "My hope is that the winning submissions will capture the most productive techniques for solving these problems, and/or provide general problem-solving techniques that would also be applicable to other types of mathematical problems."

I think your suggestions are actually complementary. Distillation of the larger networks capable of solving these problems and study of the layers could be part of the process for generating the cheat sheet.

steppion Mathematics Distillation Challenge – Equational Theories

It seems like the lift in the open-source models is being used as a proxy metric, and the core goal is a human understandable yoga [1] for approaching these kinds of equational proofs in universal algebra.

[1] https://mathoverflow.net/questions/64071/what-does-the-term-...

steppion We Stopped Using the Mathematics That Works

I spent some time in industry working on ML-based credit risk modeling. In my experience, successful shops that have a genuine interest in applying their models to practical decision making with real stakes care deeply about uncertainty quantification and decision theory. Things can get messy very fast though and the challenges faced are often too hyper-specific to one's situation to make sense as part of an academic research program. I think it's been for the best that academic research has tended to focus on the development of general algorithms intended to be broadly useful. Businesses are already well incentivized to take the best of what academia produces and try to get the decision theory right for their particular problems.

steppion Why can't you tune your guitar? (2019)

Yes, people try this. Check out dynamic tonality. It doesn't necessarily need a system. Experienced guitar players often find themselves unconsciously making little microtonal adjustments through bends and other techniques when playing leads. I found myself doing this just because it sounded better to me. I didn't even notice there was a consistent pattern until I eventually learned the math. For example I'd always want to bend minor thirds slightly sharp and bend the neck to slightly detune major thirds.

https://en.wikipedia.org/wiki/Dynamic_tonality

steppion Approaching 50 Years of String Theory

I was planning to make a similar comment. Conjecturing that some theory in the string theory landscape [0] gives a theory of quantum gravity consistent with experiments that are possible but beyond what humans may ever be capable of isn't as strong of a claim as it may first appear. The intuition I used to have was that string theory is making ridiculously specific claims about things that may remain always unobservable to humans. But the idea is not that experiments of unimaginable scale and complexity might reveal that the universe is made up of strings or something, it's just that it may turn out that string theory makes up such a rich and flexible family of theories that it could be tuned to the observed physics of some unimaginably advanced civilization. My impression is that string theory is not so flexible that its uninteresting though. There's some interesting theoretical work along these lines around exploring the swampland [1].

[0] https://en.wikipedia.org/wiki/String_theory_landscape

[1] https://en.wikipedia.org/wiki/Swampland_(physics)

steppion Mathematicians don't care about foundations (2022)

I don't think they usually make the distinction in a formal sense, but I think most are aware. The space of explorable mathematics is vastly larger than what the community of mathematicians is capable of collectively thinking about, so a lot of aesthetic judgment goes into deciding what is and what isn't interesting to work on. Mathematicians differ in their tastes too. A sense of sturdiness vs flimsiness is something that might inform this aesthetic judgment, but isn't really something most mathematicians would make part of the mathematics. Often, ones interest isn't the result itself, but some proof technique that brings some sense of insight and understanding, and exploring that often doesn't make much contact with foundational matters.

steppion Mathematicians don't care about foundations (2022)

I'd say that I care deeply about the meaning behind theorems, but just find results which swing widely based on foundational quirks to be less interesting from an aesthetic standpoint. I see the most interesting structures as the ones that are preserved across different reasonable foundations. This is speaking as someone who was trained as a pure mathematician, moved on to other things, but tries to keep up with pure math as a hobby.

steppion There is No Quintic Formula [video]

It seems like a typo where "integers" is used when the intention was to write "natural numbers". That is the solution to exercise 194 part a) which asked if the set of natural numbers is a field.

steppion There is No Quintic Formula [video]

Vladimir Arnold famously taught a proof of the insolubility of the Quintic to Moscow Highschool students in the 1960s using a concrete, low-prerequisite approach. His lectures were turned into a book Abel’s Theorem in Problems and Solutions by V.B. Alekseev which is available online here: https://webhomes.maths.ed.ac.uk/~v1ranick/papers/abel.pdf. He doesn't consider Galois theory in full generality, but instead gives a more concrete topological/geometric treatment. For anyone who wants to get a good grip on the insolubility of the quintic, but feels overwhelmed by the abstraction of modern algebra, I think this would be a good place to start.

steppion Calculus for Mathematicians, Computer Scientists, and Physicists [pdf]

Just a couple of corrections.

Let's take the Fundamental Theorem of Calculus as an example[0]:

  f'(x) = lim_{h->0} {f(x + h) - f(x)} / {h}

This isn't the Fundamental Theorem of Calculus, it's the usual definition of the derivative of a function of a single real variable. The Fundamental Theorem of Calculus establishes the inverse relationship between differentiation and integration [0].

Unless you're Ramanujan, every mathematician has spent hours banging their head against a literal or metaphorical wall (or both!)

Ramanujan was no stranger to banging his head against the wall. My impression from Kanigel's The Man Who Knew Infinity is that his work ethic and mathematical fortitude were as astonishing as his creativity. For much of his career, he couldn't afford paper in quantity and did his hard work on stone slate, only recording the results. This could make it seem like his results were a product of pure inspiration because he left no trace of the furious activity and struggle that was involved.

From The Man Who Knew Infinity:

When he thought hard, his face scrunched up, his eyes narrowed into a squint. When he figured something out, he sometimes seemed to talk to himself, smile, shake his head with pleasure. When he made a mistake, too impatient to lay down his slate pencil, he twisted his forearm toward his body in a single fluid motion and used his elbow, now aimed at the slate, as an eraser. Ramanujan's was no cool, steady Intelligence, solemnly applied to the problem at hand; he was all energy, animation, force.

[0] https://en.wikipedia.org/wiki/Fundamental_theorem_of_calculu...

steppion Dead soldiers' teeth reveal diseases that doomed Napoleon's army

From https://www.merriam-webster.com/dictionary/decimate

Decimate is a word that often raises hackles, at least those belonging to a small but committed group of logophiles who feel that it is commonly misused. The issue that they have with the decline and fall of the word decimate is that once upon a time in ancient Rome it had a very singular meaning: “to select by lot and kill every tenth man of a military unit.” However, many words in English descended from Latin have changed and/or expanded their meanings in their travels. For example, we no longer think of sinister as meaning “on the left side,” and delicious can describe things both tasty and delightful. Was the “to kill every tenth man” meaning the original use of decimate in English? Yes, but not by much. It took only a few decades for decimate to acquire its broader, familiar meaning of “to severely damage or destroy,” which has been employed steadily since the 17th century.

steppion A recent chess controversy

Your intuition's not bad. The expected value for the longest run of heads in N total flips of a fair coin is around log2(N) - 1 with a standard deviation that's approximately 1.873 plus a term that vanishes as N grows large. log2(10B) - 1 is approximately 32 and with that standard deviation, even a run of 100 in 10B flips is incredibly unlikely. For more info see Mark F. Schilling's paper, "The Longest Run of Heads" available here https://www.csun.edu/~hcmth031/tlroh.pdf.

steppion Bit is all we need: binary normalized neural networks

Yeah, I think surrogate gradients are usually used to train spiking neural nets where the binary nature is considered an end in itself, for reasons of biological plausibility or something. Not for any performance benefits. It's not an area I really know that much about though.

steppion Bit is all we need: binary normalized neural networks

It sounds like jongjong was probably using surrogate gradients. You keep the step activation in the forward pass but replace with a smooth approximation in the backwards pass.

steppion Show HN: Fast Thermodynamic Calculations in Python

Very cool project. If you haven't already, for JAX and PyTorch support take a look at the Python Array API Standard, https://data-apis.org/array-api/latest/, and see https://data-apis.org/array-api-compat/ for how to use it. If you have or can write everything in terms of the subset of NumPy supported in the Array API Standard, you can get support for alternative array libraries almost for free.

steppion Deep Learning Is Applied Topology

If I had to give a loose definition of topology, I would say that it is actually about studying spaces which have some notion of what is close and far, even if no metric exists. The core idea of neighborhoods in point set topology captures the idea of points being nearby another point, and allows defining things like continuity and sequence convergence which require a notion of closeness. From Wikipedia [0] for example

The terms 'nearby', 'arbitrarily small', and 'far apart' can all be made precise by using the concept of open sets. If we change the definition of 'open set', we change what continuous functions, compact sets, and connected sets are. Each choice of definition for 'open set' is called a topology. A set with a topology is called a topological space.

Metric spaces are an important class of topological spaces where a real, non-negative distance, also called a metric, can be defined on pairs of points in the set. Having a metric simplifies many proofs, and many of the most common topological spaces are metric spaces.

That's not to say that topology is necessarily the best lens for understanding neural networks, and the article's author has shown up in the comments to state he's moved on in his thinking. I'm just trying to clear up a misconception.

[0] https://en.wikipedia.org/wiki/General_topology

steppion Perfect Random Floating-Point Numbers

There are many situations where you have something you want to compute to within low number of units in the last place, that seem fairly involved, but there are very often clever methods that let you do it without having to go to extended or arbitrary precision. Maybe not that surprising, but it's something I find interesting.

steppion Perfect Random Floating-Point Numbers

I think the truly surprising thing is just how well floating point numbers work in many practical applications despite how different they are from the real numbers. One could call it the "unreasonable effectiveness of floating point mathematics".

steppion Probability-generating functions

https://en.m.wikipedia.org/wiki/Characteristic_function_(pro...

steppion Daniel Dennett has died

Wittgenstein himself states that the Tractatus is nonsense in its closing pages.

My propositions are elucidatory in this way: he who understands me finally recognizes them as senseless, when he has climbed out through them, on them, over them. (He must so to speak throw away the ladder, after he has climbed up on it.)

I think you may agree with the Wittgenstein of the Tractatus more than you realize. My understanding is that his main goal at that time was to show that many of the classic problems of metaphysics which plagued philosophers for centuries or more are literally just nonsense. He didn't write the Tractatus to convince regular people though, but to convince academic philosophers of his time. He earned his fame by being somewhat successful. Rather than making a logical argument for his point, I understand his aim as stimulating his audience to think things out for themselves by offering them carefully crafted nonsense that gave a fresh perspective.

I think you just have no use for the Tractatus because you're not preoccupied with metaphysical questions.

steppion LLaMA now goes faster on CPUs

Check out The Science of Programming Matrix Computations by Robert A. van de Geijn and Enrique S. Quintana-Ort. Chapter 5 walks through how to write an optimized GEMM. It involves clever use of block multiplication, choosing block sizes for optimal cache behavior for specific chips. Modern compilers just aren't able to do such things now. I've spent a little time debugging things in scipy.linalg by swapping out OpenBLAS with reference BLAS and have found the slowdown from using reference BLAS is typically at least an order of magnitude.

[0] https://www.cs.utexas.edu/users/rvdg/tmp/TSoPMC.pdf

steppion LLaMA now goes faster on CPUs

The Fortran implementation is just a reference implementation. The goal of reference BLAS [0] is to provide relatively simple and easy to understand implementations which demonstrate the interface and are intended to give correct results to test against. Perhaps an exceptional Fortran compiler which doesn't yet exist could generate code which rivals hand (or automatically) tuned optimized BLAS libraries like OpenBLAS [1], MKL [2], ATLAS [3], and those based on BLIS [4], but in practice this is not observed.

Justine observed that the threading model for LLaMA makes it impractical to integrate one of these optimized BLAS libraries, so she wrote her own hand-tuned implementations following the same principles they use.

[0] https://en.wikipedia.org/wiki/Basic_Linear_Algebra_Subprogra...

[1] https://github.com/OpenMathLib/OpenBLAS

[2] https://www.intel.com/content/www/us/en/developer/tools/onea...

[3] https://en.wikipedia.org/wiki/Automatically_Tuned_Linear_Alg...

[4]https://en.wikipedia.org/wiki/BLIS_(software)

steppion The complete story of Gödel incompleteness

There’s a wonderful book on the subject by Raymond Smullyan, of knights and knaves recreational math puzzle fame which helped make it click for me.

https://lib.undercaffeinated.xyz/get/pdf/5823

steppion A CEO's Guide to Emacs (2015)

I’ve found code generation tools a boon to my Emacs-ing because I can use them to help write Emacs lisp code for any customization I think I might need in the moment without breaking the flow of concentration towards my primary task.

steppion LLMs and Programming in the first days of 2024

I've developed a workflow that's working pretty well for me. I treat the LLM as a junior developer that I'm pair programming with and mentoring. I explain to it what I plan to work on, run ideas by it, show it code snippets I'm working on and ask it to explain what I'm doing, and whether it sees any bugs, flaws, or limitations. When I ask it to generate code, I read carefully and try to correct its mistakes. Sometimes it has good advice and helps me figure out something that's better than what I would have done on my own. What I end up with is like a living lab notebook that documents the thought processes I go through as I develop something. Like you, for individual tasks, a lot of times I could do it faster if I just wrote the damn code myself, and sometimes I fall back to that. In the longer term I feel like this pair programming approach gives me a higher average velocity. Like others are sharing, it also lowers the activation energy needed for me to get started on something, and has generally been a pretty fun way to work.