> For most of the articles Pangram flagged as written by GenAI, nearly every cited sentence in the article failed verification.
35,358 karma · joined April 8, 2009
About me: https://gwern.net/me
What I've written recently: https://gwern.net/changelog
> For most of the articles Pangram flagged as written by GenAI, nearly every cited sentence in the article failed verification.
Why do you care about LM Arena? It has so many problems, and the fact that no one would suggest using GPT-4o for doing math or coding right now, or much of anything, should tell you that a 'win rate of 70%' does not mean whatever it looks like it means. (Does GPT-4o solve roughly as many Erdos questions as gemini-3-pro...? Can you write roughly as good poetry?)
(This is the flipside of personalized pricing: it's also lower prices, as noted in some of the anecdotes, although he mostly emphasizes the potential for higher prices.)
(The right way to do this is to randomize multiple independent occasions - wait until one of you was about to call an Uber, immediately flip a coin to decide who does, each time checking you or your wife's Uber, and never both, and compare the long-run average.)
> with an apologetic confirmation dialog, possibly only after some holding period has timed out
Right. I've seen the same thing myself. They would prefer not to apologize to the customer because they changed the price, because it is in fact annoying and a bad customer experience. So the prices are surely carefully set in many ways with an eye towards not changing as much as possible.
> This enables people to construct a plausible story to explain any price.
Indeed. So you should mention one of the most plausible stories if you're going to list a bunch of them.
The most obvious possibility omitted is that your wife got the first, easy, cheap car and then Uber had to quote you a higher price to get a second car. Cars don't fall from the sky; if two people successively ask for bids, how else could it work? What if the app quoted you both the cheap price for the only car within X blocks, and you bought it before she did? Is it suddenly going to go 'oops sorry, changed my mind, it now costs twice as much'? Sounds like a very bad experience to me! More sensible to give the first person a low quote and then when - unexpected and unpredictably - someone requests something similar, quote them the higher price reflecting the sudden local micro-shortage.
That's not true. It didn't replicate and Norvig has said as much somewhere on HN, IIRC.
(I also agree with the other criticisms that this 'old vs young' setup in OP is obviously at least partially, and perhaps entirely, regression to the mean and Berkson.)
Thank you for volunteering. I look forward to your results.
Note that he doesn't quite say, when asked pointblank how much AI he used in his erroneous microbenchmarking, that he didn't use AI: https://reddit.com/r/selfhosted/comments/1qi64pr/i_got_into_...
Which explains all of it.
Kudos to /u/teraflop, for having infinitely more patience with this than I would.
What I have in mind is something far more radical. There's an idea I am calling 'log-only writing' where you stop editing or rearranging your notes at all, and you switch to pure note taking and stream of conscious braindumping, and you simply have the LLM 'compile' your entire history down into whatever specific artifact you need on demand - whether that's a web of flashcards or a blog post or a long essay or whatever. See https://gwern.net/blog/2024/rss + https://gwern.net/nenex , combined with the LLM reasoning and brainstorming 'offline' using the prompts illustrated by my poems.
I've been focused on creative writing, with poetry as my test case, to see what the bottlenecks are to truly amplifying myself through LLMs (as opposed to helping my boss automate away my job or spamming the Internet more efficiently).
I find that frontier LLMs are now there and now I can prompt for genuinely good poetry with LLMs. See https://hollisrobbinsanecdotal.substack.com/p/llm-poetry-and... / https://gwern.net/fiction/lab-animals and https://gwern.net/blog/2025/better-llm-writing
So maybe this year I can turn some attention back to PKMs and Quantified Self stuff...
I have a little rant about it - "‘Tools for thought’ winds up being a lie: there’s tools, but not much additional thought." https://gwern.net/blog/2024/tools-for-thought-failure https://www.lesswrong.com/posts/CoqFpaorNHsWxRzvz/what-comes...
(My answer, of course, is that almost all of this scutwork is well within the capabilities of a frontier LLM today. We just need to apply them.)
> Increasingly, public institutions seem to exist to manage the obsessions of a tiny number of neurotic—and possibly malicious—complainers.
Why would anyone complain about airport noise when it is ~100% guaranteed to do them no good, and almost all the benefits go to everyone else even if it somehow did anything? Just thinking like an economist here... (Indeed, if a large fraction of locals did complain about something like airport noise, that would itself be highly suspicious to me - as it would indicate an organized campaign or an issue which has become politicized in some way and is now a pretext for something else entirely like a culture war.)
And if there is something I've learned about design and problems, it's that you can have a huge problem, and you are lucky if even 1% will ever tell you.
Your website could be down, and if even 1 person takes the risk of going out of their way to tell you, you should thank your lucky stars that you have such proactive, public-spirited readers!
See also: "Theory of the Nudnik: The Future of Consumer Activism and What We Can Do to Stop It" https://gwern.net/doc/economics/2020-arbel.pdf , Arbel & Shapira 2020; https://pointersgonewild.com/2019/11/02/they-might-never-tel... (commentary: https://gwern.net/ref/chevalier-boisvert-2019); https://en.wikipedia.org/wiki/1%25_rule
If they can't predict new emergence, then 'explaining' old emergence by post hoc prediction with bizarre newly-invented metrics would seem to be irrelevant and just epicycles. You can always bend a line as you wish in curve-fitting by adding some parameters.
I don't think that paper is widely accepted. Have you seen the authors of that paper, or anyone else, use it to successfully predict (rather than postdict) anything?
Sounds like that's more than a junior or an intern that would have cost twice as much in fully loaded cost.
I would say that this graph looks a lot more extreme, actually!
I wish we didn't have that, but we have to. That demo of the theme togglebar exists to educate users, because we found that a lot of complaints came from people who were blind to the gear icon and were unaware that the control they wanted already existed. (Which is regrettable but understandable, because almost all websites provide useless controls or visual spam, in an example of 'why we can't have nice things'.)
Obviously, it's disabled on subsequent page loads. (One of a number of parts of the UI/UX we streamline using 'demo-mode', as we try to thread the Scylla & Charybdis of a cluttered but explicit UI vs a clean newbie-unfriendly UI.)
> In our new "AI/ML Product Management" class, the "pre-case" submissions (short assignments meant to prepare students for class discussion) were looking suspiciously good. Not "strong student" good. More like "this reads like a McKinsey memo that went through three rounds of editing," good...Many students who had submitted thoughtful, well-structured work could not explain basic choices in their own submission after two follow-up questions. Some could not participate at all...Oral exams are a natural response. They force real-time reasoning, application to novel prompts, and defense of actual decisions. The problem? Oral exams are a logistical nightmare. You cannot run them for a large class without turning the final exam period into a month-long hostage situation.
Written exams do not do the same thing. You can't say 'just do a written exam'. So sure, the students may prefer them, but so what? That's apples and oranges.