programjames

Maybe for an Australian billion? But in American English it would be $100 million.

I find it pretty plausible they got an 80% speedup just by making optimized kernels for everything. Even when GPUs say they're being 100% utilized, there are so many improvements to be made, like:

- Carefully interleaving shared memory loading with computation, and the whole kernel with global memory loading.

- Warp shuffling for softmax.

- Avoiding memory access conflicts in matrix multiplication.

I'm sure the guys at ClosedAI have many more optimizations they've implemented ;). They're probably eventually going to design their own chips or use photonic chips for lower energy costs, but there's still a lot of gains to be made in the software.

(Note: this is Risk).

I recently played a game with higher-level players. One player was incredibly passive; their bonus would get broken every turn, and rather than doing anything to defend it (and they had enough troops to defend it), they pulled their troops into the middle of the bonus and let everyone else take turns breaking their bonus. This could work fine in a lower-level lobby—eventually the other players might get bored and start hitting each other—but not with better players. Good players realize a couple things about them:

1. They won't retaliate, so I can knock off a few of their troops at no risk to myself.

2. They won't do anything, so any fighting among the rest of the players will effectively be a troop subsidy to them.

Since they're unwilling to help anybody, no one wants to give them free troops, and since they've demonstrated an unwillingness to retaliate, there's really no risk with hitting them over and over. So, naturally, they were the next player to be eliminated.

One thing that's nice about Risk is there's very little to the mechanics. There aren't combos to practice or build orders to memorize, mostly just an understanding of what other people want and how to negotiate. Pretty much everyone rated intermediate and above has the mechanics down: how to move troops around to not block each other, how to choke out other players, what moves are game ending for you or another player, and so on. The thing that sets apart intermediates, experts, masters, and grandmasters is almost entirely their ability to work with other players. However, since many (most?) players are "smashy noobs", lots of people rise up the ranks by just assuming everyone else is a smashy noob, and playing extremely passively to compensate. It works, until they end up in a lobby full of masters.

Just from the title I knew Keenan Crane was the advisor. He has some really good YouTube video explanations for most of the concepts:

https://www.youtube.com/watch?v=bZbuKOxH71o

This work is one of those things that feels like the completely obviously right way to do things in retrospect, and why hasn't anyone implemented this before? It helps that the authors explain it very intuitively, slowly building up the tools to run like stars.

Something I don't see mentioned is that this would be pretty useful for training a physics-based model, specifically a neural ODE. Since the mean squared error scales as O(1/N), to converge you need O(1/Loss) evaluations per point. If you were using a grid approach, the cell size would need to be O(sqrt(Loss)) in size, and the running time couldn't be better than O(1/(Loss * log(Loss))) in two dimensions. To be fair, it probably takes O(1/log(Loss)) time for each Monte Carlo simulation, so it's no worse to use the grid. But, if you go up to three or more dimensions, this method still has the same running time while the best grid method takes at least O(# grid cells) = O(1/Loss^(d/2)) time.

I've seen people with a great deal of natural ability. I'm certainly one of them. But most never got as good at maths as me, because my father began teaching me before an age I can remember. I also never got as good as some others, with arguably less natural ability, because my father didn't know what an amazing maths education looks like either—he grew up in the middle-of-nowhere, went off to college, and realized, "I really should have learned all this stuff earlier," and did his best to do so for me, but he was learning about competition maths at the same time as me. This is not to say that competition maths make an "amazing maths education", just that competition maths coaches exist, and they know exactly how to best teach students maths, while I learned mostly through trial and error.

By "unable to take advantage of it", do you mean:

- Unable to demonstrate prequisite knowledge, or

- Unable to go to a school further away?

I can get the latter one, which is why I think we shouldn't just have magnet schools, we should have free, government-run magnet boarding schools. Or, alternatively, how much would it really cost to provide a chauffeur service to kids who have demonstrated intelligence and need? If schools can provide personal aides to 1% of their population, I'm sure they have the budget to treat another 1% equitably.

And, in most places, your public school has the legal obligation to provide an education for all ability levels, including people who show up to their first year with the ability to read. Even in the United States of America, where the culture is only the bottom 20% of students deserve to learn, most school districts still have this in their bylaws (they just ignore it).

This is the same reason people don't take calculus in ninth grade, or organic chemistry in tenth. "What will you do if you run out of classes?" I don't know, learn on your own? And once you run out of learning, do your own reseach? You cannot simultaneously view school as a place to advance your child's learning, and also a place you need to hold off on their learning for. Pick one, and if you pick the latter, admit to yourself that they're not going to school, they're getting babysat.

That isn't a fair ranking. Some people are more idealist than you, and a priori want to believe others will play well. If you stick them in a lobby full of smashy noobs, they may not live long enough to update on those beliefs (within the single game). Of course, the same happens in reverse: if you go into a game assuming everyone is a smashy noob, and they're actually good at the game, you're going to end up in the losing position. The rate of mistakes changes the trembling-hand equilibria. The best players will, of course, figure out reputations for the other players as the game progresses, but calling some just 'okay' and others 'better' because they initialize the reputations to different values is not fair.

Think about this: you've probably driven a car to get from Point A to Point B before. If you existed in a society where people were constantly making mistakes, in the sense of crashing their cars several orders of magnitude more often, driving a car to get from Point A to Point B is no longer a good strategy. But it usually is a good, if not the best, move you can make right now, because people aren't making mistakes that frequently.

Here's another example: marriage (long-term relationships). Perhaps not all extra-marital affairs are mistakes, but a significant proportion of them are. If too many people are making these mistakes, leading to messy divorces, it's no longer worth it to even consider dating in the first place.

If you've ever played Risk online, this becomes obvious, very quickly. The worst players to have around are not your enemies, but the stupid ones. They'll randomly block your troops from attacking mutual enemies, accidentally bait you into attacking other players ("attack red together?" . "okay sure!" . hits three of their troops, after you hit thirty), not hold bonuses but not let you hold them either, waste all their troops on a useless endeavour, feeding the game to a third player, and so many more blunders. Sometimes, you really want to work with them, because they've tried to be nice to you, much more so than the rest of the players. Often, they fail miserably and drag you down with them. I'll take a smart enemy any day. Usually smart players don't want to be your enemy, unless you've done something to provoke them (oops... it's more fun). Even then, they'll ally with you in a heartbeat if you can take a mutually beneficial action.

Something sobering to keep in mind is that the vast majority of people have the logical and mathematical capabilities of a good ten-year-old. There are ten-year-old chess grandmasters, USAJMO qualifiers, or to be less extreme, ten-year-olds that have a solid understanding of algebra and an intuition for proofs. Most people do not, and rely on heuristics or intuitions for everything. They do not even realize you can prove something, except with "it feels or seems right". It isn't because they're mentally incapable of learning the skill, it's because they never bothered to, or didn't think it was important.

> That means that you can't pipe these things into each other, unless you can build an equivalently fast and efficient light->charge transducer (i.e. a photodetector).

These exist:

https://ultrafast.mit.edu/

There is, and people have trained purely optical neural networks:

https://arxiv.org/abs/2208.01623

The real issue is trying to backpropagate those nonlinear optics. You need a second nonlinear optical component that matches the derivative of the first nonlinear optical component. In the paper above, they approximate the derivative by slightly changing the parameters, but that means the training time scales linearly with the number of parameters in each layer.

Note: the authors claim it takes O(sqrt N) time, but they're forgetting that the learning rate mu = o(1/sqrt N) if you want to converge to a minimum:

    Loss(theta + dtheta) = Loss(theta) + dtheta * dLoss(theta) + O(dtheta^2)
                         = Loss(theta) + mu * sqrtN * C (assuming Lipschitz continuous)
    ==>     min(Loss)    = mu * sqrtN * C/2

Why not just subsume it into the other income taxes?

Tuition money doesn't fund research, so how is the GI Bill relevant?

I feel like the math term closest to that would be "consistent" not "efficient".

Why do finance bros like to call allocating all the resources to the maximal-dollar extracting process "efficient". It's not. All they're doing is turning the economy's entropy into dollar bills now, which stunts future growth.

Carcinisation in action:

     free (foss) -> non-profit -> capped-profit -> public benefits corporation -> (you guessed it)

You can train an LLM to maximize the information content bitrate. I just think most companies want to maximize "customer satisfaction" or w/e, which is why we get the verbose, bold, bullet points.

I can only find papers from the late 1980s on this, but a professor around during the 60s told me hackers were working on robotic ping-pongers way back then.

People did the conveyor belt thing in the 60s.

Feels like clickbait. People were doing similar things in the 60s at MIT. I thought by "human level robot" they meant actually human level, with a humanoid robot swinging the paddle, not a conveyor belt.

As someone born in 2003, I did this just last week when filing my tax returns.

Sure. The original normalizing flows used a fixed number of layers. Someone at UToronto recognized that, as the number of layers gets very deep, this is essentially an ordinary differential equation (ODE). Why?

Suppose you have n residual layers that look like:

x_0 = input x_{i+1} = x_i + f(x_i) x_n = output

If you replace them with an infinite number of layers, and use "time" t instead of "layer" i, you get

x(t+dt) = x(t) + dt f(x(t)) <=> x'(t) = f(x, t)

so to find the output, you just need to solve an ODE. It gets better! The goal of normalizing flows is to "flow" your probability distribution from a normal distribution to some other (e.g. image) distribution. This is usually done by trying to maximize the probability the training images should show up, according to your model, i.e.

loss(model) = product model^{-1}(training image)

Notice how you need the model to be reversible, which is pretty annoying to implement in the finite-layer case, but with some pretty lenient assumptions is guaranteed to be true for an ODE. Also, when you're inverting the model, the probabilities will change according to the derivative; since you have more than one dimension, this means you need to calculate the determinant of the Jacobian for every layer, which is decently costly in the finite-layer case. There are some tricks that can bring this down to O(layer size^2) (Hutchinson++), but the ODE case is trivial to compute (just exp(trace)).

So, turning the model into an ODE makes it blazing fast, and since you can use any ODE solver, you can train at different levels of precision based on the learning rate (i.e. the real log canonical threshold from singular learning theory). I haven't seen any papers that do this exactly, but it's common to use rougher approximations at the beginning of training. Probably the best example of this is the company Liquid AI.

Finally, this all turns out to be very similar to diffusion models. Someone realized this, and combined the two ideas into flow-matching.

-----

This is one place it's super useful to know numerical methods, but here are a couple others:

1. Weight initialization --> need to know stability analysis

2. Convolutions --> the Winograd algorithm, which is similar to ideas in the FFT and quadrature

This is why I believe a numerical methods course should be a requirement for any AI majors.

Problem 7 here: https://mathcontest.unm.edu/PastContests/2016-2017/2016-2017...

Or a Veritasium video here: https://www.youtube.com/watch?v=iSNsgj1OCLA

69% looks surprising like the answer to this puzzle:

> The numbers 1–n are randomly placed into n boxes in a line. There are n people who are each able to look into half the boxes. While they are allowed to coordinate who looks into which boxes beforehand, they are taken out one at a time to choose which half of the boxes they will peek at. The goal is for the first person to find the number one, the second person to find the number two, and so on. If any of them fail to find their number, the whole group loses. What is the probability they lose if they use the optimal strategy?

I wonder if there's a connection to regular graphs here.

It seems absolutely bonkers to me that someone would write a smart contract that lets them bleed $50m without automatically stopping after they lose the first $1/10/100k.

Next we're going to learn that winning Poker Bots with an "all in" strategy is defrauding the competition.

I thought there was a little more structure than just that (since that's been around since the 1940s).

Here's a link to the paper: https://www.nature.com/articles/s41586-024-08262-7

I really wish they tied this into representation theory, because it looks like the R-tensors are similar to representations of the permutation group, but I don't see exactly how to write them in terms of representations.