cafaxo

Yes, sorry -- I did not realize that for some reason. I removed my comment.

[Comment removed by author]

Does gradient descent really do well for deep learning when the gradient is computed with respect to the whole dataset? I assumed that the noise in SGD played an important role for escaping local minima.

I did a similar thing for Julia: Llama2.jl contains vanilla Julia code [1] for training small Llama2-style models on the CPU.

[1] https://github.com/cafaxo/Llama2.jl/tree/master/src/training

To me, the clearest way to think about signed vs. unsigned integers is that different representatives of the integers modulo n are chosen.

For example, for 8-bit signed integers we choose the representatives -128, -127, ..., 127 of the residue classes -128 + 256Z, -127 + 256Z, ..., 127 + 256Z in the ring of integers modulo 256.

For 8-bit unsigned integers, we instead choose the representatives 0, 1, ..., 255.

Mathematically, I do not see how anything is "breaking" as the article claims.

Yes, exactly.

Of course. Theoretically, the determinant answers the binary question "singular" or "nonsingular".

Numerically, such a binary answer is pretty useless. Here, we need a measure of how singular/nonsingular a matrix is relative to the numerical precision we are working with.

Edit: Sorry, I completely messed up my original answer here. A better version:

Let's say we are in a setting where we only work with integers. A matrix is invertible iff its determinant is invertible in the underlying ring. The only invertible elements in Z are -1 and 1.

So, the code is also incorrect in the integer setting. Here, we should not check for 0, but for -1 or 1.

They define a generic "is_singular" function and test it with a 2x2 matrix.

The problem with the determinant is not about performance. It is just useless for determining if a matrix is singular. The thing that gives it away is that the determinant is influenced by a rescaling of the matrix:

det(s A) = s^n det(A) where A is a n x n matrix

As an example, would you say that [[1e-10, 0], [0, 1e-10]] is singular? It has condition number 1.

From page 6, when they describe their synthetic textbook dataset:

"Consider the matrix A = np.array([[1, 2], [2, 4]]). We can check if this matrix is singular or nonsingular using the determinant function. [...]"

No. The determinant is not a suitable way to do that. A proper way to numerically measure singularity would be to compute the condition number of the matrix (the ratio of its largest to smallest singular value).

GPT-4's explanation of its optimization does not make sense to me. It writes "Instead of moving it to P, we can directly use S in the following comparisons, saving one instruction." but then proceeds to use P as if that mov had happened.

AlphaDev's optimization relies on the fact that B and C are already in the correct order. This precondition is missing from the prompt given to GPT-4. It seems that GPT-4 is hallucinating something that only resembles the correct optimization at first glance.