blt

Sure, but in games sometimes improving the average case is less important than the worst case.

What is an acceleration force?

Understandable. However, I went down that rabbit hole once, and learned that even summing a 1D array with a GPU is nondeterministic (because it is broken down like merge sort and subject to the scheduler). I guess I assumed that practitioners had fully embraced randomness due to things like that.

I agree with this part of your response: If we were to require that distributed training generates the exact same sequence of weight updates as serial SGD on a single machine, then we would need a barrier like that. However, there is a lot of research on distributed optimization that addresses this issue by relaxing the "exactly equivalent to serial SGD" requirement, including classic papers [1] and more recent ones [2].

Basically, there are two properties of ML optimization that save us: 1) the objective is a huge summation over many small objectives (the losses for each training data point), and 2) the same noise-robustness that makes SGD work in the first place can give us robustness against further noise caused by out-of-order updates.

So I think this issue can be overcome fairly easily. Does anyone know if the big LLM-training companies use asynchronous updates like [1,2]? Or do they still use a big barrier?

[1] https://proceedings.neurips.cc/paper_files/paper/2011/hash/2...

[2] https://arxiv.org/abs/2401.09135

I can see how such a phenomenon could happen at the level of a single machine, but if we're using a whole data center full of GPU machines it should be possible to spread out those spikes evenly over time. Still weird that the article implies spikiness is a fundamental property of AI workloads rather than a design oversight that can be fixed at the software level.

What is causing demand bursts in AI workloads? I would have expected that AI training is almost the exact opposite. Load a minibatch, take a gradient step, repeat forever. But the article claims that "each step of the computation corresponds to a massive energy spike."

yes

This is pretty interesting. I didn't realize so much manipulation was happening after the initial softmax temperature choice.

As a car audio enthusiast, the biggest obstacle to putting a system into a new high-tech car is bypassing the deeply-embedded infotainment system while retaining decent aesthetics and steering wheel controls. The idea of getting an electric drivetrain and new-car safety with a 90's-style blank canvas for audio is amazing.

I hope that the noise isolation and intended speaker mounting locations are good!

The design shows a fundamental misunderstanding of sheet metal. Flat sheet metal is weak. Only curved sheet metal can be strong. Designs that lack mechanical sympathy with the materials in use don't tend to age well.

Branch-and-bound is an algorithm "from the book" to me. Fundamentally very simple, provided you view the LP solver as a black box, but incredibly useful.

Sorry for being a grumpy old man, but I don't have npm on my machine and I never will. It's a bit frustrating to see more and more CLI tools depending on it.

Fully agree. Asahi is pretty good now though, and the list of missing features continues to shrink.

The prime number code hurts to read. I feel like Bob is living in a different reality than most of us.

You didn't read the article

TOSLink was kind of a silly idea because digital electrical signals would also prevent ground loops. The key is digital vs. analog, not optical vs. electrical.

Excellent choice of aircraft. The 777 is one of the greatest engineering achievements of all time.

Thank you! I used to use DuckDuckGo, but I had to switch to Google as DDG became overrun with LLM-generated results in the past 2 years. Google has some ability to reject them.

Google's new "AI overview" takes up about half of the screen on a phone and cannot be disabled. This is one of the most user-hostile choices they've ever made.

Start with the desire - we want to have an algebra of 2D affine transformations with composition and (usually) inversion.

We can represent this with the trick of "homogeneous coordinates", using 3D vectors with the last entry 1 and 3x3 matrices with the last row [0, 0, 1].

This is convenient because both mathematicians and programmers are familiar with linear transforms and matrices. It's in the comfort zone. There are many libraries.

However, it's wasteful to store all those extra 1's and 0's in memory. You can always replace the matrices with

class Affine2D { Matrix2x2 linear; Vector2 shift; };

and overload all the algebraic operators, including multiplication with 2D vectors.

Isn't this the standard way to implement cruise control in the decades before drive by wire throttles?

Not necessarily, look up "mystery meat navigation". The affliction that produced our current UI woes has existed almost as long as the WWW itself.

Thanks to LLMs, the death of stylish writing in non-literary fields is assured. I agree, many people seem to view writing as a burden.

I have only met a handful of people that can lay out a complex argument from scratch in speech alone. For most of us, writing is thinking. If you avoid writing, your ideas will remain murky forever.

this looks similar to VisPy (https://vispy.org/), are there any major differences?

The paper's name is shorter than this post title, and summarizes the result much better.

dang that's a good war story.

I was going to write a more serious rebuke, but I can't really do any better than yours

If the author is reading: In the proof, at the end of Step 6, it's confusing that the "uv" term of the integration by parts is suddenly given a range from -∞ to ∞, as if we had previously assumed x ∈ ℝ. But elsewhere in the article, including the examples, we have higher-dimensional x's. I suggest to either 1) include the full multidimensional version from the paper, or 2) explicitly mention that this is the simple 1d case, the same result holds in ℝ^n, and refer to the paper.

I don't really agree with that. Facebook was originally about mirroring your real-life social network. In the mid 00's nobody was trying to get likes from strangers on Facebook.

Instagram is closer to broadcast, but it was always closely tied to the mobile app experience and the "follower" mentality. People didn't really share links to Instagram posts in other online venues in the beginning.

Twitter was always unique. It existed before smartphones, and there was a good chunk of years where people without smartphones would read twitter posts on desktops. Its producer/consumer distribution is much more skewed, many twitter users never post. Tweets were always getting posted to places like HN, reddit, discussed in news articles, etc.

I think Twitter's (former) position as a broadcast medium à la TV, radio, and newspapers is unique among social networks. There's a reason why Twitter was the place for journalists, politicians, academics, fire departments, web service status alerts, etc.

This sentence was almost too perfect. Really says it all.