SkyBelow

The data of such a copy is nothing compared to the wider picture and the data can be used for future training, so even from a purely self interest perspective, they should be keeping the copy.

As for long term benefits, it could one day be sold as a service, once copyrights have expired on the works. We can't see it today, but that is purely the result of the law and what the law intended to do from the start, you don't see a copy unless you pay for your own.

>Despite the copyright restrictions that are forcing companies to do this, they should maintain archives that are publicly available.

Aren't the copyright laws forcing them to do this the very ones that would make such archives illegal? The books that could be in such an archive are the books that don't need to be destroyed.

And did the LLM really improve their output? They gave similar output to the LLM, so if it doesn't have a harness with tools connected to fill in the blank, it might be hallucinating details. Might not even count as a hallucination as it tries to fill in the blank with the closest looking relevant information (previous chat, maybe it has access to teams/emails and scans that for anything similar, and so on).

I've sees some chat logs where I completely empathized with the AI trying to make sense of the information it was being drip fed. Always incomplete, often incorrect.

Incorrect calculation led to some wrong value, I think electric charge. Subsequent findings showed a general trend towards a correct value. The general trend indicates that findings that were too far off were scared to be published for being wrong and contradicting existing studies, thus only small refinements were published, causing the slow drift.

Thus, the first well accepted study anchors a value and it takes much more work to get a new value established as correct after the anchor is set. Especially in a case where the experiment design itself is still solid and it was more with specific experiment (measurements slightly off, other values not quite correct).

Silly AI, doesn't it know that you have to write the test first, see it fail, and only then can you justify making a change to the Dockerfile.

I do wonder if the idea of TDD has influenced AIs to be too prone to testing even when a human would never consider it. I've seen some really silly tests, especially when it starts writing tests for things that are setup to only allow testing. Normally a comment later and it agrees it was pointless, but unless you have something in context to force it, it simply defaults to "test all the things".

This isn't just on code.

When I'm writing technical documentation, it keeps the explanations in. Same when writing non-technical documentation. When I was having it attempt to generate a Pathfinder 1e class for a Sword Dancer, it was leaving in notes about why it removes things I told it to remove/rework.

And it isn't just Claude. I've seen the same with GPT models, with Grok, with Deepseek. Each AI isn't quite the same with how it approaches this, but in every case they seem to have a strong bias to retaining information, even bad information that we want gone, so it is like they have a, dare I say, subconscious bias to retain the information. Putting a note in a comment or explaining why to not do something or something was undone is a good way to retain information while still achieving the goal (well, if you ignore the part about the human intention for the information to be gone).

This then weakens the AI in the future, as I find AI struggles with the more incorrect information. Sure, a comment saying "not X because Y" is less 'context damage' than a comment saying "X" (assuming X is wrong), but it is still a slight shift to X being present in context in some way. One off, AI's seem to perfectly handle this without issue. But after hundreds or thousands of cases build up? The attention mechanism seems unable to keep up and incorrect information flows it. This effectively creates a sort of vibe coding maximum size unless there is a human janitor cleaning up the bad information on the context stays nice and clean.

But this is all simply a feeling I get as I use AI to do different things and isn't at all backed up by any formal study.

This is based on the assumption of facts existing.

There are many studies, but each can be wrong and they can collectively show a bias. Even things of which are the most non-political of facts can have very strong biases. Look at the Millikan measurement of the electron and how in created confirmation bias and an anchoring effect on some property that has absolutely no real world significance to the things people are tribalistic about (aka, no political relevance). Now imagine the same applied to fields like economics or psychology which do have massive legal/political implications.

For a different example, ask the question if X committed crime Y. There are cases where they weren't found guilty but it is reasonable to assume they did. But being found guilty doesn't make it a fact either, as some people are wrongly convicted. Some eventually are overturned, but even if it isn't, it still isn't a fact they committed a crime.

Then there is the simple ambiguity of statements. Language generally can't support facts. It is why legalize, and programming code, and math's are effectively their own languages. For a simple example, consider the Betrand paradox(1).

>Consider an equilateral triangle that is inscribed in a circle. Suppose a chord of the circle is chosen at random. What is the probability that the chord is longer than a side of the triangle?

Is the answer 1/2, 1/3, or 1/4? Well, it is all three at once, depending upon what you meant by random. Now, imagine how this impacts things like research studies, where the randomness is much harder to quantify and there is constant pressure to p hack a result.

1. https://en.wikipedia.org/wiki/Bertrand_paradox_(probability)

Potentially worse than nothing, it allows for someone to claim they have a fix even if the fix does nothing to stop someone from becoming a victim, thus allowing for even stronger victim blaming.

Back in the day (in AI time) GitHub Copilot had Grok on the 0 github-token cost and I found it to be the best of the 0 github-token models for when my budget was out. Then they went to a multiplier that was not competitive and I haven't look back again. Been meaning too, but for personal use, Deepseek flash is so cheap I haven't felt like spending money elsewhere.

For straight generated code it'll likely need more text, but it'll still show up.

In cases where one token is extremely likely, it'll randomly be red or green and still be picked in either case as it is simply the best (or only) option. So you'll have more tokens that don't show a pattern either way (half of these cases will match and half won't, just the same as if a human wrote it). Meaning you'll need more instances where multiple tokens were all likely to see if there is a pattern. Given the check algorithm can't identify these cases, it can only judge on the overall text, so the more strict a language, the more the length requirement scales.

Where I wonder if this keeps working is in tool calls. Often, you don't take code straight from the llm, you take the results of a tool call to edit already existing code. It might be that the result of this leads to far too few signals to pick up, meaning that this only works when one does significant generation with a single model (even swapping between different models, at least by different companies, breaks this just as much as having a human write parts of the code).

Think of it like finding a loaded dice. A dice that has a slight bias in a few dozen roles is just random chance. If that bias continues after hundreds of thousands of roles, the dice is loaded. But will a code base have enough samples, especially when edits made from tool calls? I could see this being unable to detect things at the size of a reasonable PR and only being useful for massive sets of changes and only if the person behind them didn't structure their AI usage to avoid detection.

Because it won't be in the training directly. It is applied after a model generates its distribution of likely tokens, biasing each token randomly based on a random key and unrelated to any meaning of the words. So half the time, the most likely token becomes more likely and half the time it becomes less likely, and the same for every other token (when temperature is above 0).

You then look at the tokens actually picked to see how closely they follow this pattern that isn't connected to the meaning of the tokens. With enough text, you can then analyze the chance of it happening by chance verses being because the generation of the tokens was done using the algorithm, and you can save a positive result until you are arbitrarily sure. There is a chance of a false positive, but the chance of a false positive approaches the chance that the murderer happened to have fingerprints that matched your and both forensics labs happened to have mixed up the dna tests and the eye witness happened to misremember the face and your phone gps happened to glitch out and put you at the murder scene at the time of the crime all happening. It is theoretically possible only in the same sense that quantum teleporting a cat is theoretically possible.

The real question is how much text do they need for a given level of certainty and what do they check for. If they flag a positive at a p value <.01, that's a problem. If they can reasonably get a p value of < 1e-12 in only a few paragraphs of text, that is effectively no false positives (but a lot of 'too short to analyze' outcomes).

If there is enough cross over between real world knowledge engrams and abstract knowledge engram, would this allow for the jump?

One interesting (albeit sad) area which might be related are humans who are never raised with a first language. They seem to never developer abstract reasoning and even seem to lose the ability to develop it later in life. This might indicate there is some 'real world senses' -> 'direct language' -> 'indirect language' -> 'abstract abduction' hierarchy that develops, perhaps related to more real world abductions as a necessary side chain to developing abstract ones.

One of the obvious problems with this is just how difficult we find it to study intelligence purely in humans. We are measure a LLMs by a yardstick that is already known broken, but maybe this is still the right path.

I don't agree, especially for this argument.

I'm a major AI user. Coding? Sure. But language learning, some art projects, TTRPG design. I'm so far 'lost' that I have a number of AI songs on my YouTube play list. For the anti-AI people, I'm like a walking abomination.

Yet despite all this, I still am a firm believer in "if you couldn't be bothered to write it, I can't be bothered to read it" as a default stance. I feel human to human communication should remain human to human. If AI is involved, the extent should be limited to where the human to human communication couldn't be achieved otherwise and fully disclosed.

Someone might even say I'm a bit of a hypocrite. Why do I like listening to an AI song but not reading an AI blog. Honestly, I'm not sure. I'm actually a bit puzzled by it myself. Regardless of the why, I think I should be an example of someone who holds this position for reasons entirely different than those you suggest.

I don't even disagree with you in general, a lot of AI discourse does seem to have bad faith arguments put forward and I do tire of some of them. But this specific one is one of the more few that seem good faith enough we should steelman it.

>LLM responses are certainly not idempotent, as they are not even deterministic.

Isn't that more due to an optimization and not how the LLM itself runs?

Like a MoE LLM run on a single input should give the same output each time. But this is inefficient, as any given token is hitting 1 (or maybe 2 or 3) experts at a time, meaning all the other experts are doing absolutely nothing. So you upgrade it to take in multiple requests. But then any given expert can become a bottleneck, so when too many requests need a given expert, some of them are routed to a second or third best expert instead. Within the context of any single request, this looks like non-determinism, but it is still deterministic when considering the full batch.

For everyday users and everyday use cases, that is enough to treat it as non-deterministic (the harness might also send in unique data like current time which means one can never have the exact same request twice), but when talking about LLMs more theoretically, I think we need to consider they can still be ran deterministically even if that isn't as optimized.

Similar with temperature. 0 means deterministic, but anything higher with a seeded value is deterministic. If anything, temperature is us purposefully adding non-determinism to agents because they were too deterministic.

Isn't the real difficulty in how vulnerabilities can be mixed? A given vulnerability might be extremely hard to execute, but very damaging. Another vulnerability might be easier to execute, but it can't do much. But if the second one can be used to trigger the first one, you then have attack surface area of the second vulnerability with the damage of the first one.

Even if some individual case can be shown to be safe from being combined, can we identify such cases with enough confidence to justify using it reduce severity warnings?

Hmm... does this mean we could see AI worm evolution now?

In the past, a worm couldn't really evolve unless it was coded to do so, and only to the extent it was coded. But an LLM worm, which instructs the LLM to copy the instructions elsewhere, will have slight random changes made as different LLMs will not always copy it perfectly. If a counter measure is deployed, and one of this alterations allows a miscopy to survive and keeps spreading, it feels like we have hit a much more natural case of evolution of a worm than ever before.

One might even argue it is the most natural case of evolution in software because the evolution was never intentionally designed. The worm wasn't made to evolve, the LLM wasn't made with the idea of helping the worm evolve, the task trying to end the worm was done with the intent of the worm evolving. While all steps are human done, evolution wasn't intended by any of them, so if it does happen, it makes it a bit more 'natural' than every simulated evolution algorithm before it.

I feel the magic in math is that the notation is to a more universal sort of language, and the e^x captures that. e^x means that infinite series, but it means a few other things at the same time. That those different things are sometimes related, that e^2 means e*e, a simple enough operation most can understand, but that it also means that infinite series with 2 in place of x, is part of the complexity of math that makes it quite different than a classical programming language. With most programming languages, someone assigned those equivalent meaning, and while it might be important to understand why, it shows you the mind of the who designed the language. But, in math (and in true Computer Science), when you see an equivalence of this sort, whose mind are you now glimpsing into?

But, for someone just starting out (or even an expert who just wandered into a new area), this becomes a barrier to understanding. Sometimes we need the mathematical summary, what tells us the immediately answer we are interested in, perhaps even if it loses the deeper nuances that are beneficial for the experts more fluent in the language.

(And yes, all math notation is human created, so it makes my point quite a bit messier in practice.)

>These two points seem slightly conflicting. Why if a human is truly more creative than AI, must it experience "the magic and absurdity of life" if it can be creative without "combining things into output"?

I agree, yet I also think AI is extremely limited in terms of new creativity.

My guess is that humans operate on some deeper level. Much like the grokking model example where it easily trained on the training data, but only later did it really start to understand the deeper problem space and 'grok' the problem, I think LLMs are doing something similar with human thought. LLMs can throw more compute, but they are still lacking some introspection/drive that leads to the better creativity of humans. Grokking this from the training data might not be possible, or it might take some fundamentally different approach, or even an imperfection in humans that LLMs don't have (ego/drive?).

This isn't to say that LLMs could never do it, but the cost to do so might require magnitudes more scale in training times and size. Or it could be in the next model released. Or maybe they already can, but that gets trained out in favor of overall more general correctness accross all domains.

How errors add depend upon how you treat them.

Two independent task? You have a 95% chance of each task being correct, so total success is .95^2 = 90.25% correct.

Having two AIs verify the same task with each being 95% correct? That's a 5% error per each, so 1-(.05^2) = 99.75% correct.

That is assuming fully independent error rates, which is the real issue. If it is pure hallucinations, then that is an okay assumption, but if there are issues in the underlying prompt/data causing the error rates, then it becomes a very dangerous assumption and I think more human involvement is recommended to better understand and resolve the underlying data problem.

The real interesting bit is that more AI's mean the 2 in the above equations gets bigger, so the error rate gets worse or better depending upon which equation is in play. So if you are in an in-between landscape where you aren't sure there is independence or not (or how much independence between errors), it also becomes unclear if more AI makes the error rate better or worse.

I see an AI as an advanced calculator, but I see even a simple calculator as a poison to education if not strongly controlled.

Number sense is such a nebulous concept, but it captures what I've seen people struggle with. A general sixth sense about where an equation should or shouldn't go, a feeling that random numbers don't appear quite random enough and might have a pattern, or even recognizing that having a pattern signifies some relationship that otherwise has no evidence.

That entirely depends upon the cost of preventing them. At no cost, sure.

But nothing is ever free. So what is the cost.

<insert a villain who determines destroying all life is the best way to stop all future harm>

There is also the consideration of if a crime being a crime is just. Consider crimes that use to be on the book that we now consider horrible things to have outlawed or even crimes that are still on the books but not enforced because we reject them.

For example, some places us to make it a crime for kids to play pinball. Is preventing kids from playing pinball, even if it came at no cost, an unquestionably good thing?

For a sufficiently bad crime, at sufficiently low costs, preventing it is good. But those two factors are both very big questions, directly challenging the notion of "unquestionably good".

>Many developers are too focused on delivery value compared to maintenance cost.

The way this is worded feels like it leaves the blame on developers. Aren't these developers focused on exactly what they are being judged by? Shouldn't we say it is the management who is too focused on delivery value compared to maintenance cost? Is it the developer's job to guide management or the manager's job to request guidance, assuming such guidance is needed?

We need to make sure the responsibility to resolve this problem falls on those with the power to act on it, and in this, developers tend to be receiving far more responsibility to fix than power to fix.

Are court decisions universally recognized? If a court rules someone true in one area, couldn't it still be considered slander/libel in another area if another court rules it as not being true?

It is a bit like saying convicted murderer vs murderer. One is claiming the conviction for murder happened (which generally correlates to the murder happening but doesn't openly claim such) while the other directly claims the murder happens. If the conviction is later overturned, does the second claim open one up to more libability for a false claim?

I think it might be built on something more than deterministic systems. Some property that is a subset of deterministic, so all your argument still apply, but merely being deterministic is not good enough.

LLMs are what made me start considering this. Imagine a company using an LLM that was fully deterministic. All RNG was either removed or seeded in such a way that the same input (so many the seed counts as part of the input) gave the exact same output. Fully deterministic.

But such an LLM, with a slight drift in input, could still produce very different outputs. This isn't being non-deterministic, but more than the change in outputs does not naturally follow from the input. I'm thinking like how 2 double pendulums can (but not always do) greatly diverge given a very small change in their input.

So in light of that I've begun to call this new property non-chaotic. So Enterprise depends on non-chaotic systems, which are a subset of deterministic systems, and then wrangling the chaotic elements they cannot remove as much as possible.

The follow question I now have is if all LLMs are inherently chaotic, or if it is possible to have a non-chaotic LLM.

>The parents job is to keep their kids out of trouble.

I challenge that anyone believes this, and for my evidence, I would submit all the age based laws that protect children regardless of what parents do.

We have already, long ago, decided that it is the government's job to protect children, at least in cases where parents fail to do an adequate job. That's why I don't see this ending any other way. The march to total domination by the side of the government might be slow, but they already won the war around a century ago (exact timeline for laws protecting children in place of parents is a very long topic and does differ country to country, I recall hearing some places still even let kids buy alcohol if they say it is for their parents to consume).

Never really like this because it feels some arbitrary as to what counts as making a claim. Human language is filled with implications, and those implications are filled with implied claims. Yet burden or proof seems mostly used when someone makes one of those implied claims explicit, say by explicitly rejecting one.

"We should do X."

"X is not possible."

There was an implied claim that X is possible and an explicit claim that X is not possible, but often it feels that in such a scenario, it is the second individual who is deemed having a burden of proof.

This is made all the worse that there is rarely a single implied claim, but more a group of related implied claims that are also weighted. Like some combination of "data centers don't destroy the environment" and "or maybe they do, but only in amounts of destruction that are tolerable compared to human behaviors and worth the outputs gained" and even "they modify the environment, but it doesn't count as destruction, just change". And even worse (yet again), different readers of the conversation will find themselves placing different weights on these different interpretations making it near impossible to agree on what the original claim is. Thus the first person who makes an explicit claim gets all the attention which ignores all the messy implied human communications preceding it.

But why does it do so inconsistently, and sometimes even forgetting to swap back to English when it comes time to do 'normal' output? It also seems recent, as when I was using deepseek even a week ago this was very rare compared to what I was seeing yesterday. I had to start including a line asking it to stay to English because I can only speak/read English.

>and everyone else will poor and unemployed

How so? Everyone would still have their skills to provide goods and services and everyone would still have wants for other's goods and services, so an economy would still run. AI can shift the economy but it doesn't lock the entire population out of the economy. It can lock out any one group because everyone else gets the good/services of that group for cheaper from the AI, but if everyone else can't afford the AI, if the AI locks everyone out, then they trade between themselves instead. And that is the sort of 'worst case possible' outcome, not even what is likely to happen as the AI makes some things much cheaper.

I don't think that is the argument.

The argument is more "I want to do good thing X, but it will also cause bad thing Y." followed by "Wait, bad thing Y is going to happen anyways, so I might as well do good thing X so we get both X and Y instead of just X."

Viewed this way, the idea is that given the world will have bad thing Y regardless, the one impact of your choice is if good thing X exists or not, and it is better to create good thing X.

Where it becomes an issue is that there is no clear X or Y. There are many different but very related bad things, so if the one you would add is actually better or worse than what is already out there, or maybe it'll exist both ways but you make it more popular, and very subjective things to judge, so different people look at the same outcome and some agree that bad thing Y would have existed anyways and others say that no, this is a new bad thing Z that wouldn't have existed anyways.

>From where I sit it seems reasonable for Anthropic to not want their product used to create malware

Yes, I think there is a PR component to this that is often left out of this discussions.

Is this self contradictory?

Making a claim of 3 star service, without providing 3 star service evidence, isn't any different than a claim like they don't clean the dishes enough. Sure, you didn't write out the exact words, but by submitting the review, you still submitted such a claim, and so it remains a question of if you provided evidence to prove it, which gets into how deep the legal system wants the evidence verified (is a picture enough, or do you need to provide evidence the picture is of the store at the date and time relevant to your review).