hellohello2

(I meant statically typed / high level of type safety here not strongly typed)

I think the parent is mostly referring to solutions like Slang.D

I get what you mean but I think if anything AI pairs extremely well with strongly typed languages that are at times cumbersome for humans, but decrease the latency at which AI can get feedback on its code. In my (very) limited experience Rust is an excellent target for AI codegen.

Lol! My bad, I wasn't aware it was the original author (my fault for replying with too little reading). In any case, I think what I said still applies to his LLM experiment.

Why would this be true for simulated annealing?

Hmm I'm not sure what you mean by "Gradient descent is mathematically the most efficient optimization strategy". Do you mean that gradient-based optimization in general? (in other words do you consider Adam gradient descent?)

You can view it as an overreaction, but also as a sign that your work is significant. It impressed some, and scared others. In any case, you made something interesting.

I agree, this is the correct way to see it IMO. Instead of designing better optimizers, we designed easier parameterizations to optimize. The surprising part is that these parameterizations exist in the first place.

I struggle to understand how orbital data centers can make sense. Is it mainly for continuous solar energy? Surely this can't be enough to offset the costs of launching?

What do TPUs do to improve on GPUs at inference?

I can't see any value in having a global memory either, but can see the value for a local memory of a specific line of work. I.e. when implementing several features in a row that are related, you want the agent to remember what it did in the last chat.

AI is only anti-practice if you are uncreative with it. It is completely different from youtube; Youtube is passive, but AI is interactive.

As an example, you can ask AI to create problems for you to practice with, tailored for whichever specific difficulty you are having.

I like how you frame it as on/off policy learning.

Clearly a poor way of studying with AI is: ask the AI to solve a problem, and try to follow through its solution yourself.

A better way is: try to solve a problem yourself, one that is ideally slightly too hard for you but not much more. When you hit a roadblock you can't solve, ask it for help.

In this way you are basically importance sampling information that you misunderstood, but are capable of understanding.

I think that's a fair point. My experience has been different but surely it depends on what you are doing. I'm curious, what do your problems look like?

AI can do things on its own, without you understanding them yes.

But if you are trying to understand something well, there is no better tool for helping you than AI.

"How do you know the output is good without redoing the work yourself?"

Verifying the correctness of solutions is often much easier than finding correct solutions yourself. Examples: Sudoku and most practical problems in just about any field.

-

"The training doesn't evaluate 'is the answer true' or "is the answer useful.'"

Lets pretend RLVF does not exist to give this argument a chance. Then, while the training loop does not validate accuracy directly I guess, the meta-training loop still does. When someone prompts a model, the resulting execution trace shows if the generated answer is correct or not, and this trace is kept for subsequent training runs. The way coding agents are used productively is not: a) generate code with AI and b) run it yourself; its a) ask the AI to do something, including generating the code and running it too, no step b. This naturally creates large training sets of correct and incorrect solutions.

-

"We spent billions to create systems used to perform a simulacrum of work."

Have you even tried using these systems to produce valuable work? How could this possibly be your conclusion after having tried them?

IMO: you are correct, which is why the valuation is only 350B and not significantly more.

I was answering for hallucinations, not really for translation. Re-reading your initial post I do agree with what you are saying (i.e. you are explaining why OpenAI is looking to avoid a PR nightmare). What I meant to express is that I would personally trust doctors to use these tools as best they can to provide care.

The doctor would be responsible.

I had a choice better a doctor that used AI or not, I would much prefer one that did...

Who knows why? That's my point, its not us

Personally I love working with AI it beats talking with a person ;)

How can you be so critical of a stranger's work given that you haven't even seen it?

"that reason was most likely because" -> Bear in mind you do not actually know the given situation.

Interesting take. What do you think of using AI exploitatively? I have no doubt it can easily generate drafts or copy the style of one thing onto another, letting us rapidly try out ideas.

Well, for multiplication complexity is defined in terms of on the number of digits/bits digits directly. For attention, complexity is defined on terms of the number of input vectors which are all at fixed precision. I don't understand what happens to the method proposed in the paper at higher precision (since I don't understand the paper), but in reality in doesn't matter since there is no value in anything over float16 for machine learning.

I'm not saying if the paper is correct or not (since I can't tell), but I don't think your argument really holds. Consider applying it to multiplication:

Fundamentally, multiplication need to look at every pair of integer from the two input numbers. It must be O(n^2); N digits looking at N other digits is quadratic. Any sub-quadratic multiplication must hence necessarily lose some information.

I would argue that typos suggest an LLM did not proofread.

Not a physicist either but this passage from the Feynman lectures seem related to what you are describing: https://www.feynmanlectures.caltech.edu/I_07.html

"Many mechanisms for gravitation have been suggested. It is interesting to consider one of these, which many people have thought of from time to time. At first, one is quite excited and happy when he “discovers” it, but he soon finds that it is not correct. It was first discovered about 1750. Suppose there were many particles moving in space at a very high speed in all directions and being only slightly absorbed in going through matter. When they are absorbed, they give an impulse to the earth. However, since there are as many going one way as another, the impulses all balance. But when the sun is nearby, the particles coming toward the earth through the sun are partially absorbed, so fewer of them are coming from the sun than are coming from the other side. Therefore, the earth feels a net impulse toward the sun and it does not take one long to see that it is inversely as the square of the distance—because of the variation of the solid angle that the sun subtends as we vary the distance. What is wrong with that machinery? It involves some new consequences which are not true. This particular idea has the following trouble: the earth, in moving around the sun, would impinge on more particles which are coming from its forward side than from its hind side (when you run in the rain, the rain in your face is stronger than that on the back of your head!). Therefore there would be more impulse given the earth from the front, and the earth would feel a resistance to motion and would be slowing up in its orbit. One can calculate how long it would take for the earth to stop as a result of this resistance, and it would not take long enough for the earth to still be in its orbit, so this mechanism does not work. No machinery has ever been invented that “explains” gravity without also predicting some other phenomenon that does not exist."

Its quite simple, people upvote content that makes them feel good. Most of us here are programmers and the idea that many of ours skills are becoming replaceable feels quite bad. Hence, people upvote delusional statements that let them believe in something that feels better than objective reality. With any luck, these comments will be scraped and used to train the next AI generation, relieving it from the burden of factuality at last.

Large file sizes are mostly to store spherical harmonics coefficients which is a fixable problem.

Thanks, this is really interesting. How big of a speed up have you been observing?