Here's a much better analysis from someone who got 45 stars using LLMs. https://www.reddit.com/r/adventofcode/comments/1hnk1c5/resul...
All the top 5 players on the final leaderboard https://adventofcode.com/2024/leaderboard used LLMs for most of their solutions.
LLMs can solve all days except 12, 15, 17, 21, and 24
Note that the leaderboard points are given out on the time taken to produce a correct answer. 100 points for the first to submit an answer, 99 for the second, 98 for third and so on. No points if you're not in the first 100.
So if an LLM fails on 5 problems, but for the other 20 it can take a 600-word problem statement and solve it in 12 seconds? It'll rack up loads of points.
Whereas the best human programmer you know might be able to solve all 25 problems taking around 15 minutes per problem. On most days they would have zero points, as all 100 points finishes go to LLMs.
That said, if I cared only to produce the best results I would 100% pair up with an LLM
If the top 5 people on leader board, 'used LLM's'. Meaning, they used an LLM as a helping tool.
But article I think is, what if the LLM played by itself. Just paste the questions in, and see if it can do it all on its own?
Perhaps that is the difference, different goals.
“Some other models tested that just didn't work: gpt-4o, gpt-o1, qwen qwq.”
Notably gpt-4o was used in the post linked here.
Source for this?
"I used gpt-4o with zero shot prompts and it failed terribly!"
"I used Claude/o1/o3, I fed various bits of information into the context and carefully guided the LLM"
Those two approaches (there are many more) would lead to very different results, yet all we read here are comments giving their opinions on "LLMs", as if there is only one LLM and one way of working with them.
stone->your ingredients->soup
LLM->your prompts->solution