Because there's a difference between explicitly targeting some outcome and attempts at that outcome being in the training data. There would be no reasoning to explicitly aim for this rather than, say, general SVG reasoning, especially since it would be possible for targeting this over other SVG creation to undermine general SVG ability. It's not impossible that labs are targeting pelicans on bicycles specifically, but every attempt at discerning whether they are has said that similar requests for SVGs that show other things (e.g., other animals using other modes of transportation, or just other SVGs in general) show similar SVG ability.

I feel like in the beginning it was a way to highlight the fidelity of the models as they struggled to produce good latex and svgs figures. But especially now with the iterative loops that is no longer such a good benchmark. And honestly these pelicans are pretty much the same, so yeah it feels totally benchmaxxed.

Chollet, Google and Alexander Wang both admitted as such

1) Chollet stated that RL:Pre-training is 9:1.

2) Google said argon was trained on SVGs

3) Wang said that Muse Wonder was not trained on 3D models

What is most charming about this benchmark is that by now they do train on past results, and thus commit all the same errors over and over.

What exactly do you imagine is happening that is forcing them to fake the benchmark results? You think they're afraid their IPO will fail if they don't make a convincing pelican, and are hiding an illicit attempt at manipulating said result? And getting caught trying to trick people wouldn't tank the price?

This might be heresy to say on HN, but this dinky forum and its recurring comments aren't actually the center of the known universe. Faking a pelican isn't worth ruining a 200 billion dollar business. They have bigger fish to fry.

Moreover, there has been a literal massive cross-model experiment showing there is no scientific evidence for improved pelican results over any other similar prompt. If y'all could kindly consider hard evidence instead of conspiracy theories, we could move on to more interesting topics.

I got downvoted[0] heavily for even mentioning the possibility. But I'll say that even if it isn't, there are plenty of pelicans in the training set for new models. People saying "well then why does it also work for flamingos???" don't understand how LLMs work.

[0] https://news.ycombinator.com/item?id=49526966