HNHacker News
TopNewBestAskShowJobs

dawnofdusk

353 karma · joined September 8, 2022

submissionscomments
dawnofdusk··on LLMs can see and hear without any training
In actor/critic the actor and critic are normally learned, i.e., their weights are adjusted during the process. The paper is correct that their method is zero-shot, but it doesn't mention that their method is essentially equivalent to a few rounds of training but then discarding the training update.

Anyone who works with deep architectures and momentum-based optimizers knows that the first few updates alone provide large improvements in loss. In this paper the breakthrough is that computing these first few updates at test time enables one to describe the algorithm as "without training" and therefore attract hype.

dawnofdusk··on A cute proof that makes e natural
You can also allow it to be multi-valued and consider a principal branch when needed, the same we do when we discuss roots of monomials in algebra. The two situations are identical (as they must be, because logarithms generalize roots).
dawnofdusk··on David Tong Lectures on Theoretical Physics
Cool that his lecture notes are now books! The lack of thermodynamics/statistical mechanics is an unfortunate oversight, however, otherwise one could feasibly use his books exclusively for an undergraduate-level degree program in physics.
dawnofdusk··on I should have loved biology too
I do quantitative biology now, although my background is in theoretical physics. Biology is fascinating, but ultimately there is a cultural divide between the scientific "language" used in biology and the scientific language of e.g., engineers, physicists (very famously described in "Can a biologist fix a radio?" https://www.cell.com/action/showPdf?pii=S1535-6108%2802%2900...)

I do find the author's point weird. "I thought high school biology was just memorizing facts, but I began to appreciate it when I read some pop science books and went scuba diving." So the only problem for the author was the topic of the classes, not the style. Why shouldn't one have the same problem with high school physics ("it's just about boring ramps and pulleys"), etc.? Personally I find the style to be a more important distinguishing factor, in that biology is much less quantitative than other science disciplines. Instead the author's problem is that biology should be even less quantitative and more literary or poetic...?

Ultimately science journalism/popularization is not the same thing as science. High school science classes (try to) teach the latter not the former.

dawnofdusk··on A cute proof that makes e natural
It doesn't break down, one just needs the complex logarithm. If you ignore complex numbers it breaks down in both cases. If you allow complex numbers it works in both cases.
dawnofdusk··on A cute proof that makes e natural
>The problem with this approach is that, since we were already introduced to exponents and logarithms in algebra but via different definitions, it always left this unanswered question in my head about how we knew these two definitions were the same, since everyone quickly glossed over that fact.

They just shouldn't be taught in algebra. There, one is thinking about oh how do I extend the definition of exponentiation from integers to real numbers. But thinking about continuous extensions of an integer function to the real line is really something that should be saved for calculus.

dawnofdusk··on A cute proof that makes e natural
Actually historically John Napier discovered the logarithm before Jacob Bernoulli computed e by his compound interest formula. IMO it would be more fruitful to teach the special functions exp(x) and ln(x) just as functions, which is more historical, with the important and unique property that they turn multiplication into addition and vice versa. The fact that exp(1) is some irrational number in between 2 and 3 is just a fun fact at this point, and the connection to compound interest is an even more fun fact.

My assumption is that some schools think the number e should be introduced much like the number pi. Except the number pi has a much more natural definition, relating to circles, and appears in important formulas like the area of a circle. Obviously it would be possible to first introduce the special functions sin(x), cos(x), etc. for which it is a fun fact that there is some irrational number between 3 and 4 such that sin(pi) = 0, but it's clear that this would be silly.

For the number e, however, this approach is not so silly, as what is important and useful are indeed the functions exp(x) and ln(x). The constant e itself does not appear in any natural or intuitive formulas, but only in connection with the function exp(x) evaluated at x=1 (or x = i*pi, in the famous formula, which should not really be taught at the high school level IMO).

dawnofdusk··on A cute proof that makes e natural
The sequence (1+1/n)^n can be seen as a natural thing to study (indeed, tautologically, because we know that e is a natural thing to study) by various viewpoints.

The important aspect however is that the compound interest interpretation of this sequence is not only unnatural but wrong, in the sense that it starts from the wrong guess that a rate r compounded n times should be something like a rate r/n. It is unnecessarily confusing to teach such young students historical mistakes: bad students won't understand the mistake and good students will be bewildered. In both cases students come away thinking of e as a sort of "mystical" thing.

If I had to teach the compound interest, one starts by considering with a yearly interest rate r, your principal grows after m years by (1+r)^m. Now, can we find an equivalent monthly interest rate r2? To do so, we must solve (1+r2)^12 = (1+r), which requires logarithms, at which point you note immediately that r2 =/= r/12, which is perhaps unexpected. Now, it might be natural to ask what about for continuous compounding? Then, we study (1+f(n))^n as n goes to infinity, where f(n) is some function of n. We know it should decrease for large n, and the binomial expansion lets us guess (for integer n) that it should be f(n) = c/n for some constant c. Now, we are ready to compute the value of this limit.

The important part is that in studying compound interest one needs certain analytical tools, such as logarithms and asymptotic analysis, beforehand. It does not make much sense, as you say, to skip over all the intermediate steps and introduce the constant e as the solution to some unmotivated and unnatural formula relating to compound interest, but this is indeed what American schools do. In my experience, very few American high school students understand or remember (if they were ever taught it) the identity a^x = e^(x * ln a), and the concept of exponentiation is generally not well understood.

dawnofdusk··on A cute proof that makes e natural
A nice approach (which is also in Po Shen Loh's preprint) is that this limit can be seen as the Euler scheme for solving the differential equation f' = f. This is related to your viewpoint about Lie theory... maybe slightly more digestible for high school students.
dawnofdusk··on A cute proof that makes e natural
The arxiv preprint linked in this is really good. I'm American so I got my education on e from the compound interest limit which isn't natural at all, as Loh points out. Why should it matter how many times I "split up" my compounding?

IMO exponentials should just not be taught at all without basic notions of calculus (slopes of tangent lines suffice, as Po Shen Loh does here). The geometric intuition matters more than how to algebraically manipulate derivatives. The differential equation is by far the most natural approach, and it deserves to be taught earlier to students as is done apparently in France and Russia.

dawnofdusk··on Most people don't care about quality
Obviously not a universal statement. Clearly an American doesn't care about the difference between Domino's and better pizza, but Italians or Europeans do. Just because your standards are low doesn't mean everyone's is.
dawnofdusk··on Derivative at a Discontinuity
I don't think it makes sense to allow derivatives of a function f to have a larger domain than the domain of f.

>which is why they’re defined via limits

They're defined via studying f(x+h) - f(x) with a limit h -> 0. But, your example is taking two limits, h->0 and x->1, simultaneously. This is not the same thing.

dawnofdusk··on Inferring neural activity before plasticity for learning beyond backpropagation
Are there any results about the "optimality" of backpropagation? Can one show that it emerges naturally from some Bayesian optimality criterion or a dynamic programming principle? This is a significant advantage that the "free energy principle" people have.

For example, let's say instead of gradient descent you want to do a Newton descent. Then maybe there's a better way to compute the needed weight updates besides backprop?

dawnofdusk··on A solution to The Onion problem of J. Kenji Lopez-Alt (2021)
Would be really interesting if you could reverse engineer the model which yields 1/phi as the correct answer. Evidently for some non-uniform measure on the onion you could do it. What about for considering the onion as a half-ball? (Although if you're cooking it really is primarily the thickness that matters.)
dawnofdusk··on Show HN: Rust library for numerical integration of real-valued functions
The Nyquist sampling theorem is of course proved by considering a Fourier transform, which is given by an integral, so the relation to integration in general should not be surprising.
dawnofdusk··on New textbook teaches students about matrix methods and their real world apps
The content of the book looks really nice, but I'm not sure why one needs a book that explicitly aims to "engage students" and is also targeted at upper-level undergraduates or graduate students. Surely students at this level have a sufficient level of self motivation to get by... would have preferred to see the same content targeted at first year undergraduates.
dawnofdusk··on Reactive HTML Notebooks
>when I'm just trying to do some exploratory data analysis

Isn't the point to have a unified platform to do exploratory data analysis which can then easily be published? It's not for throwaway Jupyter notebooks.

I think there's great value for an alternative to Jupyter notebooks that aren't just throwaway. The UX being really trash right now is something which can be improved IMO. The question is whether this setup is better than the JSON madness in Jupyter notebooks... I'm leaning yes personally.

dawnofdusk··on Diffusion models are evolutionary algorithms
>I do not see the same concept in diffusion models.

I believe,

Population size = noise dimensionality

Population diversity = noise magnitude

dawnofdusk··on Scaling Laws of Optimization
Didn't know Francis Bach had a blog. For anyone interested in the theory of machine learning, he is definitely one of the leaders.
dawnofdusk··on An n-ball Between n-balls
I think our geometric intuition could never be good for a high dimensional object. Consider a sphere in 3D. It's represented by the points which satisfy x^2 + y^2 + z^2 = 1. Because there are only three coordinates, knowing the value of one of them greatly reduces the possibilities for the other two. For example, if I know z is close to 1, then my point is close to the north pole (x and y are both close to zero).

However, if I have the n-sphere x_1^2 + x_2^2 + ... + x_n^2 = 1, knowing the value of x_1 gives me very little information about all the other coordinates. And humans' interaction with the geometry is reality is somewhat limited to manipulating one coordinate at a time, i.e., our intuition is for built on things like moving our body linearly through space, not dilating the volume or surface area of our bodies.

dawnofdusk··on Nobel Prize in Physics awarded to John Hopfield and Geoffrey Hinton [pdf]
There is no nobel prize for biology
dawnofdusk··on Nobel Prize in Physics awarded to John Hopfield and Geoffrey Hinton [pdf]
Agree completely, being in this field.

However, it is weird for the committee to give a prize for theoretical physics without an experiment. It is doubly weird when they already made this "mistake" in 2021 with Parisi, who was the odd one out among the geophysicists, and are giving another prize in spin glass/stat phys... why?

dawnofdusk··on Nobel Prize in Physics awarded to John Hopfield and Geoffrey Hinton [pdf]
Giorgio Parisi's prize proves the committee gives prizes for theoretical discoveries nowadays. This year's prize is more proof.
dawnofdusk··on LaTeX Style Guide for EE 364B [pdf] (2014)
Bertsekas' textbooks of course, which I think are very well-written and a pleasure to read.
dawnofdusk··on Too much efficiency makes everything worse (2022)
The author is an academic, it is important to give proper credit for ideas within reason. Same reason I call F = ma the law of Newton and now the law of my high school physics teacher, even though I learned it first from him.

The reason I have this quibble is because the author says things like

>you should consider building formal (mathematical) bridges between results on overfitting in machine learning, and problems in economics, political science, management science, operations research, and elsewhere

If we are appropriately modest and acknowledge the fact that overfitting is well-studied by statisticians (although, obviously not in the context of deep neural networks), it seems kind of ridiculous to make statements like, economists and political scientists should consider using statistics?

dawnofdusk··on Too much efficiency makes everything worse (2022)
The author is a very sharp individual but is there a reason he insists on labelling overfitting as a phenomenon from machine learning instead of from classical statistics?
dawnofdusk··on Everything you need to know about Python 3.13 – JIT and GIL went up the hill
But the performance motivations for removing the GIL seem at odds with this. I feel like the subset of Python users who care about the GIL and the subset who are "non-programmers" are entirely disjoint.
dawnofdusk··on Meta outage
Messenger is also down. For some people this is an important means of communication (many young people I know in the US). People may either be in a panic or otherwise cannot coordinate sufficiently the plans they may have had to get to the polls.

That being said this year's primaries have got to be historically uncontested that it could not matter less.

dawnofdusk··on Why do tree-based models still outperform deep learning on tabular data? (2022)
Paper seems interesting but I don't like the question title. I think the answer to the question would just be that tabular data is not fully in the "big data" regime yet so there is no reason a priori to expect deep NNs to do better. Factor in computational simplicity of tree-based models and I think the deck is stacked against deep learning from the start.
dawnofdusk··on Why do tree-based models still outperform deep learning on tabular data? (2022)
The paper offers a reason why NNs working for tabular data would be good:

>Creating tabular-specific deep learning architectures is a very active area of research (see section 2) given that tree-based models are not differentiable, and thus cannot be easily composed and jointly trained with other deep learning blocks.

Here is a second reason, from the paper

>Impressed by the superiority of tree-based models on tabular data, we strive to understand which inductive biases make them well-suited for these data.

which is a great reason, because understanding the inductive biases of different learning/regression techniques gets us closer to a more general understanding of how to encode inductive biases in a generic learning algorithm.

← PreviousPage 3 of 5Next →