anotherpaulg

I've long been fascinated by AI's ability to do the reverse: generate photos with lots of highly relevant content when the prompt includes a location. Terrain, plants, buildings, landmarks, coastlines and lots of details are included.

Here's an example [0] for "Riding e-scooters along the waterfront in Auckland". The iconic spire is correctly included, but so are many small details about the waterfront.

I've been meaning to harness this into a very-low-bandwidth image compression system. Where you take a photo and crunch it to an absurdly low resolution that includes EXIF data with GPS, date/time. You then reconstruct the fine details with AI.

Most photos are taken where lots of photos are taken, so the models have probably been appropriately trained.

[0] https://chatgpt.com/share/680d0008-54a0-8012-91b7-6b1794f485...

In aider, instead of “ultrathink” you would say:

  /thinking-tokens 32k

Or, shorthand:

  /think 32k

Mostly Gemini 2.5 Pro lately.

I get asked this often enough that I have a FAQ entry with automatically updating statistics [0].

  Model               Tokens     Pct

  Gemini 2.5 Pro   4,027,983   88.1%
  Sonnet 3.7         518,708   11.3%
  gpt-4.1-mini        11,775    0.3%
  gpt-4.1             10,687    0.2%

[0] https://aider.chat/docs/faq.html#what-llms-do-you-use-to-bui...

I just finished updating the aider polyglot leaderboard [0] with GPT-4.1, mini and nano. My results basically agree with OpenAI's published numbers.

Results, with other models for comparison:

    Model                       Score   Cost

    Gemini 2.5 Pro Preview 03-25 72.9%  $ 6.32
    claude-3-7-sonnet-20250219   64.9%  $36.83
    o3-mini (high)               60.4%  $18.16
    Grok 3 Beta                  53.3%  $11.03
  * gpt-4.1                      52.4%  $ 9.86
    Grok 3 Mini Beta (high)      49.3%  $ 0.73
  * gpt-4.1-mini                 32.4%  $ 1.99
    gpt-4o-2024-11-20            18.2%  $ 6.74
  * gpt-4.1-nano                  8.9%  $ 0.43

Aider v0.82.0 is also out with support for these new models [1]. Aider wrote 92% of the code in this release, a tie with v0.78.0 from 3 weeks ago.

[0] https://aider.chat/docs/leaderboards/

[1] https://aider.chat/HISTORY.html

Aider author here.

Based on some DMs with the Gemini team, they weren't aware that aider supports a "diff-fenced" edit format. And that it is specifically tuned to work well with Gemini models. So they didn't think to try it when they ran the aider benchmarks internally.

Beyond that, I spend significant energy tuning aider to work well with top models. That is in fact the entire reason for aider's benchmark suite: to quantitatively measure and improve how well aider works with LLMs.

Aider makes various adjustments to how it prompts and interacts with most every top model, to provide the very best possible AI coding results.

The graph in the article about Quasar Alpha’s coding skill is taken from my tweet [0]. It shows QA’s results on the aider polyglot coding benchmark [1].

QA seems to be a skilled coder, and is very fast.

Aider supports Quasar Alpha as of v0.81, released last week.

[0] https://x.com/paulgauthier/status/1907996176605220995

[1] https://aider.chat/docs/leaderboards/

Llama 4 Maverick scored 16% on the aider polyglot coding benchmark [0].

  73% Gemini 2.5 Pro (SOTA)
  60% Sonnet 3.7 (no thinking)
  55% DeepSeek V3 0324
  22% Qwen Max
  16% Qwen2.5-Coder-32B-Instruct
  16% Llama 4 Maverick

[0] https://aider.chat/docs/leaderboards/?highlight=Maverick

I actually get asked for screencasts a lot, so I made recently made some [0].

The recording of adding support for 100+ new coding languages with tree-sitter [1] shows some pretty advanced usage. It includes using aider to script downloading a collection of files, and using ad-hoc bash scripts to have aider modify a collection of files.

[0] https://aider.chat/docs/recordings/

[1] https://aider.chat/docs/recordings/tree-sitter-language-pack...

Gemini 2.5 Pro set a wide SOTA on the aider polyglot coding leaderboard [0]. It scored 73%, well ahead of the previous 65% SOTA from Sonnet 3.7.

I use LLMs to improve aider, which is >30k lines of python. So not a toy codebase, not greenfield.

I used Gemini 2.5 Pro for the majority of the work on the latest aider release [1]. This is the first release in a very long time which wasn't predominantly written using Sonnet.

The biggest challenge with Gemini right now is the very tight rate limits. Most of my Sonnet usage lately is just when I am waiting for Gemini’s rate limits to cool down.

[0] https://aider.chat/docs/leaderboards/

[1] https://aider.chat/docs/faq.html#what-llms-do-you-use-to-bui...

It does have fairly low adherence to the edit format, compared to the other frontier models. But it is much better than any previous Gemini model in this regard.

Aider automatically asks models to retry malformed edits, so it recovers. And goes on to produce a SOTA score.

Gemini 2.5 Pro set the SOTA on the aider polyglot coding leaderboard [0] with a score of 73%.

This is well ahead of thinking/reasoning models. A huge jump from prior Gemini models. The first Gemini model to effectively use efficient diff-like editing formats.

[0] https://aider.chat/docs/leaderboards/

Yup. It’s a complex enough set of in/excludes that I think that would get unwieldy for my use case.

Details here:

https://github.com/Aider-AI/aider/blob/main/scripts/blame.py

Again, nice work on your tool. I’ll spend some more time trying to harness it for my need.

This is great. I do this sort of git-blame accounting to track how much code is written by AI versus humans in each release of my app.

My "blame script" has been slowing down as the repo size increases. I was just about to add caching, like you have.

Have you thought about adding the ability to limit the stats based on a set of file patterns? Perhaps like this, where the file follows gitignore conventions?

  git-who table -include-file <fname>
  git-who table -ignore-file <fname>

I tried to quickly add this functionality but unfortunately I don't know go.

Here’s my write up about the unique ways that aider uses uv as an installer.

https://aider.chat/2025/01/15/uv.html

Offering these install methods dramatically reduced the number of GitHub issues from users with conflicted/broken python environments.

You can use —no-auto-commits or see these docs for all the git settings:

https://aider.chat/docs/git.html

Thanks for trying aider! Sorry to hear you had trouble with the install. Installing Python cli tools is tricky, which is why I recommend using the uv-based installer.

The docs do list a variety of other install methods:

https://aider.chat/docs/install.html

And there is a faq about running aider from source:

https://aider.chat/docs/faq.html#how-can-i-run-aider-locally...

GPT-4.5 Preview scored 45% on aider's polyglot coding benchmark [0]. OpenAI describes it as "good at creative tasks" [1], so perhaps it is not primarily intended for coding.

  65% Sonnet 3.7, 32k think tokens (SOTA)
  60% Sonnet 3.7, no thinking
  48% DeepSeek V3
  45% GPT 4.5 Preview <===
  27% ChatGPT-4o
  23% GPT-4o

[0] https://aider.chat/docs/leaderboards/

[1] https://platform.openai.com/docs/models#gpt-4-5

Good point!

This is easy to picture if you imagine widely spreading out the equipment used for the eraser experiment. If the signal hitting the screen and idler hitting one of the detectors are space-like separated events... the OP's explanation no longer seems to apply.

This is a very interesting, novel take on explaining the delayed choice quantum eraser. I think I can summarize it as follows:

The signal photon hits the screen, which is a measurement. The entangled idler's wave function is thereby constrained by that measurement, influencing the probabilities of later detecting it at each of the D1-4 detectors. It's not that the fate/erasing of the idler changes the already committed path(s) of the signal. It's that the measurement of the signal constrains the subsequent detection of the idler.

Two other high quality discussions of eraser experiments are by Sean Caroll [0] and Sabine Hossenfelder [1]. Like the OP, both Sabine and Sean demystify/debunk these experiments.

These three discussions all use different language to explain the outcome, which is clearly predicted by the QM math. Sean's article includes the gist of the math.

[0] https://www.preposterousuniverse.com/blog/2019/09/21/the-not...

[1] https://www.youtube.com/watch?v=RQv5CVELG3U

Here are the current docs for changing the thinking token limits.

https://aider.chat/docs/llms/anthropic.html#thinking-tokens

I'll make this less clunky soon.

I try not to let perfect be the enemy of good. All benchmarks have limitations.

The Exercism problems have proven to be very effective at measuring an LLM's ability to modify existing code. I receive a lot of feedback that the aider benchmarks correlate strongly with people's "vibes" on model coding skill. I agree. The scores have felt quite aligned with my hands-on experience coding with most of the top models over the last 18+ months.

To be clear, the purpose of the benchmark is to help me quantitatively assess and improve aider and make it more effective. But it's also turned out to be a great way to measure the coding skill of LLMs.

Using up to 32k thinking tokens, Sonnet 3.7 set SOTA with a 64.9% score.

  65% Sonnet 3.7, 32k thinking
  64% R1+Sonnet 3.5
  62% o1 high
  60% Sonnet 3.7, no thinking
  60% o3-mini high
  57% R1
  52% Sonnet 3.5

Claude 3.7 Sonnet scored 60.4% on the aider polyglot leaderboard [0], WITHOUT USING THINKING.

Tied for 3rd place with o3-mini-high. Sonnet 3.7 has the highest non-thinking score, taking that title from Sonnet 3.5.

Aider 0.75.0 is out with support for 3.7 Sonnet [1].

Thinking support and thinking benchmark results coming soon.

[0] https://aider.chat/docs/leaderboards/

[1] https://aider.chat/HISTORY.html#aider-v0750

Thanks for letting folks know about aider's /copy-context command.

To add some more detail, aider has a mode/UX that is optimized for "copy and paste" coding with LLM web chats. The "big brain" LLM in the web chat does the hard work, and a cheap/local LLM works with aider to apply edits to your local files.

There's a little demo video in the link above that should give you the gist.

"Improving forecasting ability" is a central plot point of the recent fictional account of How AI Takeover Might Happen in 2 Years [0]. It's an interesting read, and is also being discussed on HN [1].

... [T]hese researchers are working long hours to put themselves out of a job. They need AI agents that can think ahead, so engineers train agents to forecast. They hold out training data before 2024, instructing models to ponder for hours to predict events in 2025. Then, they apply the same trick as before, distilling pondering into a gut reaction. Forecasting ability is a broad foundation. The researchers build specialized ML research skills on top of it, training U3 to predict the results of every ML paper and ML experiment ever recorded.

[0] https://www.lesswrong.com/posts/KFJ2LFogYqzfGB3uX/how-ai-tak...

[1] https://news.ycombinator.com/item?id=43004579

For AI coding, o3-mini scored similarly to o1 at 10X less cost on the aider polyglot benchmark [0]. This comparison was with both models using high reasoning effort. o3-mini with medium effort scored in between R1 and Sonnet.

  62% $186 o1 high
  60%  $18 o3-mini high
  57%   $5 DeepSeek R1
  54%   $9 o3-mini medium
  52%  $14 Sonnet
  48%   $0 DeepSeek V3

[0] https://aider.chat/docs/leaderboards/

Every metric has limitations, but git blame line counts seem pretty uncontroversial.

Typical aider changes are not like autocompleting braces or reformatting code. You tell aider what to do in natural language, like a pair programmer. It then modifies one or more files to accomplish that task.

Here's a recent small aider commit, for flavor.

  -# load these from aider/resources/model-settings.yml
  -# use the proper packaging way to locate that file
  -# ai!
  +import importlib.resources
  +
  +# Load model settings from package resource
  MODEL_SETTINGS = []
  +with importlib.resources.open_text("aider.resources", "model-settings.yml") as f:
  +    model_settings_list = yaml.safe_load(f)
  +    for model_settings_dict in model_settings_list:
  +        MODEL_SETTINGS.append(ModelSettings(**model_settings_dict))
  

https://github.com/Aider-AI/aider/commit/5095a9e1c3f82303f0b...

It's "total" tokens, input plus output. I'd guess more than two-thirds of them are input tokens.

When I was mostly just using Sonnet I was spending ~$100/month on their API. That included some amount of bulk API use for benchmarking, not just my interactive AI coding.

Not using shebang, but I've recently been using uv as an "installer". It's hard to package and distribute python CLI tools with complex dependencies. I used uv in a couple of novel ways:

1. I copied their `curl | sh` install script and added a `uv tool install --python python3.12 my-tool` line at the end. So users can now install my CLI tool with a nice curl-pipe-sh one-liner.

2. I made a tiny pypi "installer" package that has "uv" as its only dependency. All it does is `uv tool install` my CLI tool.

Both methods install the CLI tool, python3.12 and all python dependencies in their own isolated environment. Users don't need to manage python virtual envs. They don't need to even have python installed for the curl-pipe-sh method.

I now get far fewer GitHub issues from users who have somehow mangled the complex dependencies of my tool.

I wrote it up with more detail and links to code:

https://aider.chat/2025/01/15/uv.html