HNHacker News
TopNewBestAskShowJobs

salamo

408 karma · joined November 27, 2022

https://blog.lukesalamone.com
submissionscomments
salamo··on Multi-agent chatbot murder mystery
> It's like if GTA V ended your game as soon as you jaywalked, and showed you a moralizing lecture about why breaking the law is bad.

There was a game called Driving in Tehran which was exactly that. If you speed or crash, you get fined. If you hit someone, it tells you "don't play games with people's lives" and then exits entirely.

https://www.youtube.com/watch?v=hFDYvlm7geU

salamo··on Glest – The free real-time strategy game
I spent a ton of time on Glest growing up. The fact that it could be modded with simple XML files made it really accessible and was a gateway for me to get into programming. So even if seems a bit dated today Glest will always be special to me.
salamo··on The brain makes a lot of waste. Now scientists think they know where it goes
Switching my smart lights to red works pretty well for me, although from the street it definitely looks like something unholy is happening in the apartment.
salamo··on My weekend project turned into a 3 year journey
Reminds me of Dan Luu's article I read this morning: https://danluu.com/sounds-easy/

I think part of why we underestimate difficulty is, it's hard to see anything but the happy path. The complications are harder to anticipate. And for each of the solutions to a problem, we also only think of the happy path to solving it. And so on.

salamo··on 20x Faster Background Removal in the Browser Using ONNX Runtime with WebGPU
They are probably two different use cases. Parallelizing with web workers could be faster for algorithms that do a lot of branching (minimax comes to mind) but if you can vectorize (matmuls for example) then GPU probably dominates.
salamo··on 20x Faster Background Removal in the Browser Using ONNX Runtime with WebGPU
"Neural networks are the second best way of doing just about anything." ~ John Denker

It's an old quote that, although not 100% accurate anymore, still sums up my feelings quite nicely.

salamo··on Climate Zones
Seems like the old -> new temperatures are the same for each city.
salamo··on Barcelona will eliminate tourist apartments
Hostels and hotels are strictly better in my opinion. I stayed in a hostel in Barcelona. A group of us went to a football game and had a great time.
salamo··on Artificial Intelligence: A Modern Approach, 4th ed
Reading through the first edition (published 1995) is actually pretty interesting. At this point it's basically a historical document. The section on neural nets contains this gem from John Denker: "Neural networks are the second best way of doing just about anything."
salamo··on Fast Crimes at Lambda School
It's really too bad. Higher ed needs disruption. The problem is, a responsibly-run university is a slow/medium growth enterprise, not an overnight unicorn.

The Vanderbilts and Carnegies of today could do a major public service if they wanted to. Use their names and hire the best researchers in their respective fields and pay them well.

salamo··on Show HN: ThreadQuilt: AI-Free Thread Aggregator
This is cool. I clicked "deep learning models" and the results were ok, but some of them were out of date. It would be great if all of the threads were recent (within the last week or so).
salamo··on Vectorized K-Means Clustering
Thank you!
salamo··on NumPy 2.0
When I started out I was basically stumbling around for code that worked. Things got a lot easier for me once I sat down and actually understood broadcasting.

The rules are: 1) scalars always broadcast, 2) if one vector has fewer dimensions, left pad it with 1s and 3) starting from the right, check dimension compatibility, where compatibility means the dimensions are equal or one of them is 1. Example: np.ones((2,3,1)) * np.ones((1,4)) = np.ones((2,3,4))

Once your dimensions are correct, it's a lot easier to reason your way through a problem, similar to how basic dimensional analysis in physics can verify your answer makes some sense.

(I would disable broadcasting if I could, since it has caused way too many silent bugs in my experience. JAX can, but I don't feel like learning another library to do this.)

Once I understood broadcasting, it was a lot easier to practice vectorizing basic algorithms.

salamo··on Can language models serve as text-based world simulators?
Yes, I've noticed it this week as well. Simple errors like misspelling variable names.
salamo··on From grep to SPLADE: a journey through semantic search
I've had pretty good success with BM25 + stemming, or even easier, BM25 with trigram tokenization. If the index isn't too big, the whole search can be done client-side and is lightning fast.
salamo··on AI Search: The Bitter-Er Lesson
If the game was small enough to memorize, like tic tac toe, you could definitely train a neural net to 100% accuracy. I've done it, it works.

The problem is that for most of the interesting problems out there, it isn't possible to see every possibility let alone memorize it.

salamo··on AI Search: The Bitter-Er Lesson
Search is almost certainly necessary, and I think the trillion dollar cluster maximalists probably need to talk to people who created superhuman chess engines that now can run on smartphones. Because one possibility is that someone figures out how to beat your trillion dollar cluster with a million dollar cluster, or 500k million dollar clusters.

On chess specifically, my takeaway is that the branching factor in chess never gets so high that a breadth-first approach is unworkable. The median branching factor (i.e. the number of legal moves) maxes out at around 40 but generally stays near 30. The most moves I have ever found in any position from a real game was 147, but at that point almost every move is checkmate anyways.

Creating superhuman go engines was a challenge for a long time because the branching factor is so much larger than chess.

Since MCTS is less thorough, it makes sense that a full search could find a weakness and exploit it. To me, the question is whether we can apply breadth-first approaches to larger games and situations, and I think the answer is clearly no. Unlike chess, the branching factor of real-world situations is orders of magnitude larger.

But also unlike chess, which is highly chaotic (small decisions matter a lot for future state), most small decisions don't matter. If you're flying from NYC to LA, it matters a lot if you drive or fly or walk. It mostly doesn't matter if you walk out the door starting with your left foot or your right. It mostly doesn't matter if you blink now or in two seconds.

salamo··on Solving Probabilistic Tic-Tac-Toe
I think there is. In this case we have to clarify that an optimal strategy can still lose due to random chance, so the optimal strategy will only be clear after some large number of games.
salamo··on What If We Recaption Billions of Web Images with LLaMA-3?
I guess it depends on the purpose. As a query rewrite, these cover much more of the physical scene, but lack accuracy and specificity.

A couple of issues. First, the model completely botches the "Trike" brotherhood to "Tike", which also drops the meaning of the symbol. It's a tricycle. The seat isn't a sidecar, it's behind the driver's. Some of the numbers for the racecars also seem to be hallucinated.

It also doesn't seem accurate to describe the rewritten captions as having a "richer" vocabulary as the authors do. It's certainly more verbose and captures a lot of details about the setting, but lacks information density for the subject of the photo. You can't make up for low specificity with more high-probability words. Sometimes you need a long-tail word.

For example, the Western Kingbird caption. It's not just a grey and yellow bird. If the model doesn't know what a Western Kingbird is, you're not going to be able to draw a Western Kingbird no matter how verbose your description is.

salamo··on Terence Tao on proof checkers and AI programs
Not a mathematician, but I can imagine a few different things which make proofs much more difficult if we simply tried to map chess algorithms to theorem proving. In chess, each board position is a node in a game tree and the legal moves represent edges to other nodes in the game tree. We could represent a proof as a path through a tree of legal transformations to some initial state as well.

But the first problem is, the number of legal transformations is actually infinite. (Maybe I am wrong about this.) So it immediately becomes impossible to search the full tree of possibilities.

Ok, so maybe a breadth-first approach won't work. Maybe we can use something like Monte Carlo tree search with move (i.e. math operation) ordering. But unlike chess/go, we can't just use rollouts because the "game" never ends. You can always keep tacking on more operations.

Maybe with a constrained set of transformations and a really good move ordering function it would be possible. Maybe Lean is already doing this.

salamo··on ARC Prize – a $1M+ competition towards open AGI progress
They claim that the average score for humans is between 85% and 100%, so I think there's a disagreement on whether the test is actually too hard. Taking them at their word, if no existing model can score even half what the average human can, the test is certainly measuring some kind of significant difference.

I guess there might be a disagreement of whether the problems in ARC are a representative sample of all of the possible abstract programs which could be synthesized, but then again most LLMs are also trained on human data.

salamo··on ARC Prize – a $1M+ competition towards open AGI progress
This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and hard to synthesize.

If I can make one criticism/observation of the tests, it seems that most of them reason about perfect information in a game-theoretic sense. However, many if not most of the more challenging problems we encounter involve hidden information. Poker and negotiations are examples of problem solving in imperfect information scenarios. Smoothly navigating social situations also requires a related problem of working with hidden information.

One of the really interesting things we humans are able to do is to take the rules of a game and generate strategies. While we do have some algorithms which can "teach themselves" e.g. to play go or chess, those same self-play algorithms don't work on hidden information games. One of the really interesting capabilities of any generally-intelligent system would be synthesizing a general problem solver for those kinds of situations as well.

salamo··on AI Appears to Rapidly Be Approaching Brick Wall Where It Can't Get Smarter
What transformers have taught us is that gobbling up the whole internet can get you pretty far, but it doesn't change the fundamentals. Hallucinations still are trivial [1] to elicit. On the other hand, transformers are a software breakthrough that made training on the whole internet even feasible. Other methods (RNNs and LSTMs) were simply too slow. Transformers are a new hammer for an internet-sized nail.

If I can make an analogy to chess engines, a type of AI so old that people don't call it AI anymore [2], if we had started out by applying transformers to available chess games, we would never be able to create an engine that could reliably beat a grandmaster. Predicting the next chess move based on some representation of the game and its relationship with previously-seen positions only gets you so far. Deep Mind tried [3] this.

One route forward, which doesn't require a fundamental breakthrough, would be synthetic data. To stick with chess, we could create huge synthetic datasets of a huge amount of chess games, and simply continue training as we were. However, this is very expensive, and can be tricky if the synthetic distribution isn't the same as the real one. And the end result is still going to require incredible computation power to use.

What an LLM really lacks is the ability to search. And search is really important [4]: without it you'll probably need orders of magnitude more data and orders of magnitude larger models. At a fundamental level, search means we don't just make a snap judgement about e.g. the likely next token, but, as Daniel Kahneman would say, "thinking slowly" about what to do/say next.

Assuming we will not have search is how you get projections like needing 100 GW power plants to supply dedicated training data centers. The human brain uses 12 watts. 100 GW is the energy requirements of all human brains on the planet. And yet, if we put all human brains on the planet together, our collective capabilities would be outclassed by a chess engine I can run on my phone.

[1] https://arxiv.org/pdf/2406.02061

[2] https://en.wikipedia.org/wiki/AI_effect

[3] https://arxiv.org/pdf/2402.04494

[4] https://www.youtube.com/watch?v=ceCg90Q9N6Y&t=1200s

salamo··on Mitsubishi robot solves Rubik's Cube in 0.305s
Hopefully the military will aim to minimize civilian casualties.
salamo··on Some notes on influenceering
I decided a long time ago that my main audience will be myself in the future. A blog post is the output of a project which at a minimum forces me to organize my thoughts on something important, and usually involves getting my hands dirty to understand something at a deeper level. I don't care about SEO because I'm writing for myself, not robots. That said, I feel for people trying to monetize because I know I never could.
salamo··on The NSA advises you to turn your phone off and back on once a week
Not only can you not access your files, but you are voluntarily restarting your phone. It seems pretty secure to me!
salamo··on X is justifiably slow (2022)
One day I checked top and noticed that I had a python process was using 60 GB of RAM. Slightly worrying. Turns out I had a jupyter notebook open from a week ago that was casually consuming 60 GB with a huge dict.

So part of the reason may be that developers tend to have access to hardware which isn't representative of a typical user. I probably could've written a more clever algorithm at the time, but why bother for a one-off?

salamo··on Detect AI-generated images instantly
Reminds me of GPT Zero, which gained a ton of attention despite marketing promises which are fundamentally impossible to keep.
salamo··on LZW and GIF explained
I found out the other day that zstd supports dictionary compression [0]. It's a pretty neat concept: you can "train" a dictionary on your corpus of text, and use this dictionary to compress subsequent documents. This is really useful if you have a lot of small documents with a similar distribution of characters. You only have to transmit the dictionary once over the wire once, and then clients can get great compression ratios after that.

For example, suppose you have a lot of chess PGNs you'd like to store [1]. If you compress each PGN separately, you won't get nearly as high of a compression ratio as if you stored them all in the same file and compressed that file. But storing games individually is more convenient. Here are the numbers:

Original PGN file: 780 bytes

Default zstd compression: 508 bytes (35% decrease)

With dictionary: 376 bytes (52% decrease)

I'm thinking about ways to compress at an even higher ratio. Chess PGNs are a great candidate for this task because they have a huge amount of redundant and repeated text.

[0] https://news.ycombinator.com/item?id=35917279

[1] https://en.wikipedia.org/wiki/Portable_Game_Notation

salamo··on Golden Gate Claude
To fix your car, step 1 is to visit the Golden Gate Bridge, step 2 is to fix your car.
← PreviousPage 2 of 4Next →