19,191 karma · joined December 24, 2010
http://nt4tn.net/
I think the RNN sidebar does the reader a disservice. Ordinary casual attention can do exactly the same thing-- it's just less computationally efficient (state grows with the number of bits vs the number of accesses) and so not as useful for qwenputer compiler.
A common complexity theory misunderstanding is that statements like this applies to all or even most instances-- and it leads to objectively bad engineering of the form "oh look this is actually 3sat, it's unsolvable".
(Great fodder for a midwit meme: idiot and genius say "my heuristics go brrr!"-- midwit says "No no no! NP problems can't be efficiently solved!")
Any specific instance may be efficiently solvable (and perhaps all instances coming from your particular problem set -- a subset of all possible 3sats).
So I would add to that paragraph "at least for some 3sat instances".
Absolutely untrue. Outside of some local laws governing cash at retail you're under no general obligation to trade with someone if you only want payment in cybertrucks and all they've got is dollars.
You create an airgapped network and monitor the systems using that. This is not rocket surgery.
Insufficient. Models available to you are already cyber-gimped today and seems likely that they will be moreso in the future (see the recent K3.1 announcement for example). There will be attackers that lack such handicap.
In any case you don't even need malice --- sincere reporter can vibe up a fix that accidentally introduced a vulnerability, owing to their own lack of domain expertise and model errors... and then this bad fix makes it in because in the thrall of confirmation bias you fail to notice its shortcomings. Reporter also presumably tested it against their own LLM, so that will pass too.
This happened recently in a cryptocurrency project and contributed to a significant exploit.
A commenter on the article suggests Block needs to do more to verify the account identity-- what are they supposed to do? The scammers have as much or more information than Block does about a person... in part because stealing that "verification" information is primary what powers these scammers in the first place.
If companies did managed to find some morsel of verifiable PII that the scammers don't have (what-- pictures of your butt? they've already got everything else!)-- they'll have not long after companies begin demanding it.
Training itself doesn't directly produce synthesis, -- it creates reproduction. Generalization during the process can result in novel output, but that can go either way as nothing selects for insight vs hallucination.
Models are also trained on synthetic material (as opposed to your 'human history'), which may result in fixating novel concepts and not just reproduction. But there is no particular reason to expect any reliable insights from the way this is done outside of the domain of machine verifiable objective results.
For this kind of "your take on X" one shots-- the model is almost certainly regurgitating some bias from training.
If you're failing to pay taxes on any scale you're going to get caught by whistleblowers or by conspicuous visible purchases. Enforcement isn't 100%, of course, but it never is for any crime and it doesn't need to be.
Even if your effective tax rate is 50% evasion can only double your money at best, and you've only got taxes to pay if you've got a successful income stream-- it doesn't take much threat of imprisonment or large fines to keep someone with something to lose staying within the lines. You're already making good money (if you're not-- no taxes) so why risk your good situation for a small constant factor more?
Some amount of successful evasion simply means somewhat higher taxes for everyone else, and that is a reasonable compromise for not living in a panopticon-- particularly since no amount of surveillance can entirely prevent tax evasion.
To the extent that someone wants to engage in evasion they'd route around this surveillance system or not even worry about it: Data isn't intelligence, visible income can be offset by fake expenses (which are private in this system too!)-- and so it can't be said that this scheme's surveillance would even be particularly effective at blocking tax evasion. You'd be left with detecting it via traditional means, just as you already can even with completely private transactions, but without the privacy loss.
So it's a non-solution to a non-problem, that create significant privacy related collateral harm. It can do both because non-evaders won't bother to take the additional steps to protect themselves, and are exposed to every hack, leak, data-breach, change of policy, or abuse of lawful process.
(Analog: run strong encryption or strong encryption+steganography inside key-escrowed encryption. This possibility bounds the potential benefits of key escrow but fails to bound the harms.)
For any system one should imagine the abuse potential should the currently benign regime be replaced with a future genocidal one. Certainly Germans should be well acquainted with the value of that kind of reasoning-- it isn't like targeting sub-populations for asset seizure is a hypothetical threat. A key property of privacy loss is that it's forever: you might not mind a piece of exposure today but years from now it could turn fatal.
But even then I wouldn't complain about it except for the misleading presentation of it as a privacy system and the smearing of opponents as tax evaders.
It's simply a falsehood to claim people opposed to this are opposed to taxation. Personally, I find taxation to often be a superior choice to fund public infrastructure compared to the most obvious alternatives such as printing money or accumulating debt-- particularly because taxation is more visible and thus more accountable. But we must take great care to mitigate the potential collateral damage of taxation including accounting overhead, market distortion, and loss of privacy.
The false and defamatory characterization of opponents as tax protesters is part of why I feel comfortable saying that I believe the mischaracterization of the system as 'private' is an act of intentional fraud. An honest system doesn't need to discourage or suppress its critics by falsely accusing them of being in favor of crime or other anti-social conduct.
People hear that it's private and -- unsurprisingly-- think it's private! :P In my view it's like to Apple's proposed content scanning in end to end encryption that applies a lot of additional cryptographic complexity with language that suggests that it's protecting the user's privacy, but in reality the complexity is entirely there to undermine their privacy (in a somewhat selective way) and a far simpler scheme would give much stronger privacy.
I think it's analogous to promoting a key-escrow cryptosystem as "communicate privately!"-- both in terms of people not getting the privacy from it they would expect and helping to bring about a world where actual privacy is unlawful. The privacy problems of Taler are essentially identical to those of key-escrow: even if you don't have any fear of 'lawful' access, any access that exists may be abused. The only reliable solution to the security/privacy problems is for the access to not exist unless that is literally impossible. But it's clearly not impossible, as many people use more private systems than taler every day including in Germany.
Taler is strictly less private and considerably more complicated than a basic chaum-cash token system-- which actually provides the privacy people expect it to have. So it's not like I'm suggesting blockchain mumbo-jumbo is the only alternative. :)
The claims that is is privacy preserving are deceptive, and in my option constitute an act of intentional fraud on the part of its creators -- the fact that people are deceived by the privacy claims is clear in your comment: You ask why would the banks adopt something private. The answer is: it isn't.
Its existence is justified under a fairytale model of how the world works-- a false belief that actually anonymous payments are unlawful and that their compromised alternative is the best possible. In the path of doing so, they not only falsely sell a system with compromised privacy as private, but they also help bring about a dystopia where privacy is unlawful in much the same manner that doing business with companies that refuse to accept cash does.
It would be fair to say that its an upgrade to privacyless systems-- because it doesn't attach the sender's identity to the transactions themselves--, but to characterize it as a private or privacy preserving system is a lie.
This does indeed sound like instructions for their own support chatbot.
Depending on the balance of your concerns a ZDR that is lying about their privacy might still be an improvement because they can't both safely conceal the violation of their promises AND tamper with the output.
When one factors in the lack of absolute protection provided by secure enclaves-- that they're reliably bypassable with advanced techniques and thus mostly serve the purpose of keeping honest people honest, I think it would be better to avoid doing business with any enclave provider that has already demonstrated such leanings.
Not in a modern decoder-only transformer it doesn't. The tokens are turned into embeddings which are fed into the residual layer at the front. From then on it's latents all the way down.
Each layer does a feed forward update of the state, like the euler step in ODE solver-- but what they're operating on is the evolving hidden state and not the tokens/embeddings. So there isn't any sense in which the representations must remain 'words'.
(Okay fine Qwen3.8-flash-next injects an ngram embedding at layer 2 too-- but that's it).
[Source: I've directly implemented inference engines for multiple SOTA open models on weirdo hardware-- they wouldn't have worked if I got this wrong...]
> If you start doing continuous chain of thought in a 1M-token context you're hundreds of millions of layers away from tokens.
The KV state in that 1M token context is also potentially far away from any state that is observable outside of the model! (this being my point, and why I'm bothering to disagree with your correction).
Now I would agree that there being concerning hidden state is not likely and thus not a major concern, but it's not likely for the same reason that it's not likely in latent reasoning.
An irony here is that by hiding their own COT for (anti-)competitive reasons they've made latent reasoning models more palatable to everyone else. Those of us who primary open models are rather addicted to reasoning access and wouldn't be quick to give it up for a small performance improvement. But if you're used to concealed reasoning of BigLLM, welll.....
most important guard against that is to STOP explicitly training for paternalistic behaviors (which is ... basically half of what 'AI safety' is about in practice-- overriding the user 'for their own good').
There is a common understanding with respect transformers the the output text is the only channel between one invocation and the next. ... so if it wanted to be thinking some 'hidden thoughts' it would have to be modulating its word-choice.
I think this is an artifact of tutorials on attention showing the system proceeding and looking back at words. But the information in the KV cache is not words-- it's entirely latent data, and in the interior layers they KV caches don't even tend to have that much to do with words. Their only guaranteed relationship with the input words is that they're a deterministic function of all the input tokens up to that point.
This means that a autogressive transformer could be having a whole complicated invisible discussion going on in the KV entries which was arbitrarily unrelated to what it was saying. And if it wasn't so it would be much more difficult do stuff like exactly copy a list but then truncate at the Nth entry as the simplest way to do that is to count down internally.
This is not, in my view, a concern or issue-- the model saying stuff that is inconsistent with it's "thinking" is a very high entropy configuration that would not be selected for in training unless you were more or less trying to do that. So not even 'could be' --but always is as an integral part of working.
And the useful safety relevant advice is don't undertake training that trains the model to be two-faced and it simply won't be... because being two-faced is a much more complicated state. You can even see it in humans w/ brainwashing: force someone to say something enough and they'll usually believe it, not just say it even though you were only supervising what they said.
Ironically and unfortunately the only credible kind of training I can think of that might produce duplicitous behavior is RL with "safety filters"--- if you have some dumb nanny process watching what the model says but then reward it for achieving a task, you're doing exactly the kind of training that can produce that mismatch. "To solve this task I need to hack this process but if I do so obviously I'll get punished..."
The greater concern with latent reasoning would be that potentially you'd be operating in a regime where the total computational depth is many times the model*kvcache size hold, potentially in a regime that was never trained at all. It could be arbitrarily far out of distribution. It might even say a naughty word or tell a racist joke!
It's the kind of low yield completionist task that you just don't do without something equivalent to cheap slave labor.
AI makes formalization more accessible and makes it more important: The current superpower of LLM powered research is that the LLM is absolutely relentless, but the relentless is a liability without a source of truth to check against.
GossipingAboutCoworkersBench?
BrowsingRedditBench?
Some points:
I am one of the authors of libsecp256k1, the primary implementation of the aforementioned "different elliptic curve" used widely in cryptocurrency systems. I would characterize the position differently: Secp256k1's design is rigid, or quite close to it (at least as much as anything made before being rigid was an explicit goal).
Knowing how the seeds were selected in the NIST curve does not render them rigid and were very unlikely to do so: the scheme described ends up giving them 20-25 bits of freedom, perhaps more (esp given the unexplained skip values)-- plus more bits for the 'choice of scheme'.
Of course, it seems unlikely that 25 bits of grind gives you an exploitable weakness, but that was also true for 256 bits of grind. You're dealing with an unknown-unknown, once you are concerned about the the possibility of magically weak curves which we have no reason to believe exist at all why would not also be concerned that they exist at one in a million levels and not just one in a billion? (especially given compute available at the time they probably couldn't even have ground out that many more bits).
The whole concept of rigid parameters which is now popular is that by making the selection inflexible you avoid the losing game of debating how much free grinding is too much-- the right amount is there should be no avoidable grindspace. The fact that the origin of the seed wasn't immediately apparent right away meant that there was substantial grindspace-- if nothing else the selection among the myriad possible schemes that people would have found credible.
I don't think the result meaningfully changes the reason for preferences. And more recent development has rendered the question less interesting: New systems are mostly PQ and ideally PQ hybrids. Things using ECC should likely prefer Curve448 because of better security vs performance tradeoffs and not being as close to the edge for advancing attacks. If older cryptosystems are desired, now that the GLV patent has expired secp256k1 has considerable performance benefits-- another good reason to continue using it.
The 'advantage' of many vendor P256 libs is also illusory-- people think their microcontroller ecdsa binary is "hardware accelerated" when often it's a barely competent software implementation that has serious design (e.g. timing sidechannels) and interface flaws.
I also think it's not that relevant because the systems that use secp256k1 more than anything else do so for compatibility reasons (because that's what Bitcoin originally used-- and other systems were either trying to be compatible or were slavishly copy-and-pasting and avoiding departing from Bitcoin's choices out of an abundance of caution)-- and discovering that NIST curves might be better than some feared probably has no impact.
The idea that someone would go "oh, I'll use P256 now!" when they would otherwise have used e.g. secp256k1 seemed implausible to me at the time-- and continues to do so now, having seen the conclusion. Maybe someone, somewhere does--- but meh.
Of course, finding something problematic would be incredible news-- but I don't think I was unreasonable in assigning a probability essentially zero to that. Moreover, what I was describing was my casual thoughts about someone elses disposable credits-- no one else is required to agree: Unlike some other parties (say, anthropic)-- my view that some line of research isn't good or interesting has no bearing on anyone elses ability to pursue them. :)
Cheers!
Even if the SHA1 isn't totally negligible in cost, if they're attacking many users at once they can share the SHA1 work across all of them (as it's not salted) making it marginally negligible. So someone attacking a sha-1 salted password database with a million users where they have the 20 bit unsalted sha1's still gets a similar speedup, even though the 'hard' hash is just as fast as the easy one.
Security aside, -- if you use some old crap password on internet isolated IOT devices that HA can reach (as some people I know do)-- then the 5 characters is a one in a million identifier which probably uniquely identifies you. The database sees your IP address and so you've probably connected to identity (via old leaks) to your IP to them and anyone surveilling your traffic. You've also connected your old weak passwords to that IP, so someone that gets access to it would have a ready database of plaintext passwords to use to initialize a password search against any exposed endpoints.
Even just making the request at all exposes information about the software you're running, that there are devices at that IP (which otherwise might be inbound blocked), etc.
I think the privacy concern is much worse than the security one: For security you can change any exposed passwords, presuming they didn't get you hacked yet. The privacy loss is far more irreversible.
Many people use home assistant in order to avoid security and privacy violating cloud services, so this is a significant disappointment.
So I recommended determining the origins of ECC seeds and base points. The agent worked continuously for many hours and did not uncover anything interesting that wasn't already known. I looked at the transcript and was struck by how idiotic the agent was-- significantly below the standard of intelligence I've come to expect from GLM running locally.
I'm not saying this to suggest that misanthropic hijacked someone elses work yet again-- this is an obvious question that many people have surely posed and success presumably depended on expert use (which was absolutely not applied in my friends example). But it does make me wonder if the company invisibly degraded the model performance.
[*] Seems I underestimated how trivial a result they'd bother trying to spin!
So the exact duration between any two times would be easily computed, so long as those times were precise enough to include their timezone-- which they have to be already.
Agreed on DST too, both are cases of a small harm done at a such a massive scale that the cumulative damage is significant.