2,466 karma · joined January 15, 2014
Note, there are many ways to represent words visually on computers that look identical
You can carefully select which pseudorandom number generator (prng) you use to be able to id text of a certain length. I expect there is some performance characteristic you have to manage since you're doing this on every inference, but once you do that it doesn't change the output in any meaningful way (the prng is still a statistically valid prng, it just happens to let you check if the output used that prng)
The point is that you can do this simply by swapping to a different RNG, which isn't noticeable to the end user, and while it changes the output, it's not any different from how using a different seed or being lumped in a different batch will change the output.
^ excerpt:
> So then to watermark, instead of selecting the next token randomly, the idea will be to select it pseudorandomly, using a cryptographic pseudorandom function, whose key is known only to OpenAI. That won’t make any detectable difference to the end user, assuming the end user can’t distinguish the pseudorandom numbers from truly random ones. But now you can choose a pseudorandom function that secretly biases a certain score—a sum over a certain function g evaluated at each n-gram (sequence of n consecutive tokens), for some small n—which score you can also compute if you know the key for this pseudorandom function.
Do you think this is an impossible task and we shouldn't try to solve it? Or do you think it's doable and that some ai detectors might be better than others?
Many users are not smart enough to realize that the transcription step is where the ai (watermarks) were necessarily injected.
When you have a lot of free users the business demands that you serve them with the best cheap model you can build
And time spent building that may provide dividends (eg OpenAI has very good RL and reasoning) but it might take resources away from the larger model training
(I have no inside knowledge, so please consider this to all be speculation)
https://x.com/polynoamial/status/2083478171975082334
As a complete guess, it seems like they tested hundreds to thousands of problems with a relatively low per-problem budget
--
The linked tweet from Noam Brown at OpenAI reads:
> And yes we did try other major problems without success. Sadly no Millennium Prize problems (yet).
> But also, we didn’t spend a lot on each problem. It’s possible to push test-time compute much further.
I discovered the reset, but it took a minute and I expect given the way I read being able to explicitly walk the high watermark back a bit would be nice (maybe that's doable if you select text? I didn't try!)
The format is nice, I love standard ebooks
I opened Moby Dick but didn't see how to listen to it on my phone, not sure what the status of the "listen" effort is.
I listened to some books from standard ebooks using eleven labs, but I'm really excited about a better implementation
First, this is about not just a job or career, but someone's identity. And we must have compassion that they are losing something that is fundamental to who they are. To them, it doesn't just feel like they're losing it, it is being taken by these companies that have so often acted in ways we despise
And that is a tragedy! And there are so many tragedies like this that will happen regularly as the technology advances
But my second reaction is one of this great shared experience. I studied math. Many of my friends are mathematicians. And it is now possible for anybody to access mathematical insights or think deeply about strange conjectures and theorems that were previously incomprehensible to anyone who hadn't at least studied mathematics in college
I don't know if 3b1b's Grant Sanderson considers himself a mathematician or science communicator, but I think of him as both. And while I expect he is someone with the mental capacity to find and prove new things in the world, I am grateful for the time he spends instead understanding things at a fundamental level and explaining and celebrating those concepts with his audience.
Not every mathematician can be or wants to be _that_ kind of mathematician. But for the moment, it is enough for me that higher mathematics is more accessible than ever.
And with some trepidation, I predict that the "traditional" job of a mathematician will change (of course it will). And it will change in big obvious ways and also subtle little ones. How will it change, though?
Importantly, math is actually going to be one of the fields where the fundamental things we thought we knew are shaken, because in the next five years we are going to start to see connections between things that were previously considered completely separate.
And so one way the job will change is that anyone who discusses math regularly will need to learn new things.
To me, this is exciting. It's almost like finding a bunch of new dinosaur fossils that fundamentally reshape our understanding. There's going to be a lot of work to do!
Could they do it? Sure, but to what end? It would make more people hate them and feel even more "take our interesting work." Pitching it as a useful tool just makes more sense on all levels
As an aside, there's this idea in math that when you create a new field you shouldn't solve all the easy problems - you need to entice other people to learn about the field!
I expect there's some element of that here. It's much better for OpenAI and Anthropic if their users are the ones discovering and writing up the results of the AI solving hard math. Look at the high school and college age students who have become ai power users and potentially learned how to use git to contribute ai generated solutions
(Related: I believe Terry has also gotten all of the subscriptions gifted to him)
https://xenaproject.wordpress.com/2026/07/20/human-mathemati...
Agreed on linear vs n log n, ofc, and streaming considerations might also be relevant
> Consider the following real problem, one of the steps in scikit-learn’s gradient histogram boosting algorithm:
> You have a large array of floating point numbers.
> You want to assign them to the integer range 0-254, spread out evenly.
Naively I would consider sorting the initial array and then using something like `batched` from itertools to chunk them into the 255 buckets - binary search will give you a bunch of random accesses, and sorting can be cache-oblivious (eg efficient for arbitrary data sizes)
But I'm somewhat concerned I don't fully understand the underlying problem being solved with this step, so I might be misunderstanding the intended result
> 2. Use a compiled language to write a Python extension.
It's fine to be grumpy about languages, but python is the de facto standard in many industries, and a lot of people don't get to choose.
So, yes, asking "how do you speed up computational python code?" is a good, fine, normal question.
I just fundamentally disagree with this line of thinking:
> even a subscription is better than royalties
Royalties are a great business model that allows more creators to create. You can dislike it, but I genuinely don't know that this game would exist with a different business model, and I want it to be easy to make art (or other things!)
And while there were a bunch of interesting side effects of piracy (people who pirate more music were bigger fans of music and spent more later - correlation/causation questions obvs), I would strongly prefer that the easy thing to do (ie cheapest/lowest friction to get started) be legal
Eg much is not known about dinosaurs. Many things cannot be found in fossils (obligatory xkcd: https://xkcd.com/1747/)
How do you communicate what is unknown or what can't be known?
My contention is that people can build more ambitious games that are more likely to make their creators rich if they use Unreal, and if the creators also think that they should use Unreal and if they don't they shouldn't.
All of the people who I was reading and listening to and watching before AI are all still creating. Many of them are even creating new things or new types of things
I imagine breaking into the creative industry is "harder" today than it was five years ago and much harder than it was ten years ago etc, but in some ways everything is still so new.
This dismissal of creativity or people's drive to create or anything created after "the ChatGPT moment" is simply wrong and lazy thinking
You can continue to enjoy the books and articles etc
They are just also used to create a new thing
In the picnic example, the picnickers can't use the lawn anymore.
As an aside: as somebody who lives in a dense area, I also would stay off the lawn. There's a utilitarian element where you have to rotate which lawns are used, and avoid using them when wet etc so as to maximize their utility. The picnickers should find a lawn that's open.
Unreal is like venture capital or a book advance (or the equivalent in music record deal)
Can you self publish? Sure, of course, have fun. But if you want the support and infrastructure of a company that understands the business of books, you take a deal and it is just like this: if a bunch of authors get book advances, that is generous to the ones who are unsuccessful, and they can only do that _because they capture the upside of those that are successful_.
Without that, you don't get advances for anyone.
So the point I'm making here: unreal provides variance reduction for all game publishers and yes that disproportionately benefits the ones who make under a million. But they're the ones who need the help!
And in exchange, if you're one of the lucky few, you pay a shockingly reasonable 5% in perpetuity.
If unreal cost money up front, would this have been built? No.
Unreal is saying: hey, we contributed to 1/20th of your success, because you could not have done this without us.
Thus, in the event that you're extremely successful, yes, you'll owe unreal a million dollars. But that's only because you made 20mm and keep 19mm for yourself.
That's an incredible bargain.
Thus, he always includes a relaxed mode to let someone practice without any stress. Incidentally, he realized that some people only ever play in the relaxed mode!
There is more to it than this, but much of the cost structure around subscriptions etc is specifically designed to allow for that experimentation.
There are good cynical takes, here, too. At the current model costs I don't need to optimize my expenses, but that could change if it climbs eg above 30% of my salary^
Note: this is an easy thing to prove ROI on. If I'm writing 5-6x more code and reviewing commensurately more code, and those PRs are better-tested and get us to shipping quality features faster, this is easy to justify and we are not that price sensitive
I can't quickly tell what all you have archived^, but I have some friends who are academic historians who might be interested in certain categories of work (and could help verify some esoteric languages) - is it possible to search by region or language?
Have you reached out to any types of historians WRT the project? It seems like some PhD students might be able to find some projects in this work etc
^ when I looked at the timeline https://sourcelibrary.org/timeline, I got an error
My understanding is that distillation uses specific probing queries to extract model weights in some meaningful sense.
There's also the element of "how one uses model outputs can be in accordance with or against the TOS"
(This is true of any website! How I access and use the info on the website can be in accordance with or against the TOS)
Both of those things: doing something against TOS, using specific queries to illegitimately gain access to model weights - that's why calling it adversarial and stealing feel fair.
I didn't (and don't) understand your point, but in an earlier post you made some claim about it being Anthropic - you can't steal from Anthropic. The above is the only reason I can think one might believe that
Was hoping someone else had proposed it though. It's a good thought