Comment by altairprime | Hacker News Reader

Comment by altairprime | Hacker News Reader

Parent Full thread

We have nothing to protect sites against scrapers except to make it more expensive for everyone’s, unless privacy-compromising or authority-trusting methods are on the table.

Making you pay time, power, bandwidth, or money to access content does not significantly impede your browsing, so long as the cost is appropriately small. For the user above reporting thirty seconds of maxcpu, that’s excessive for a median normal person (but us hackers are not that).

If giving your unique burned-in crypto-attested device ID is acceptable, there’s an entire standard for that, and when your device is found to misbehave, your device can be banned. Nintendo, Sony, Xbox call this a “console ban”; it’s quite effective because it’s stunningly expensive to replace a device.

If submitting proof of citizenship through whatever attestation protocol is palatable is okay, the Anubis could simply add the digital ID web standard and let users skip the proof of work in exchange for affirming that they have a valid digital ID. But this only works if your specific identity can be banned, or else AI crawlers will just send a valid anonymized digital ID header.

This problem repeats in every suggested outcome: either you make it more difficult for users to access a site, or you require users to waste energy to access a site, or you require identifiable information signed by a dependable third-party authority to be presented such that a ban is possible based on it. IP addresses don’t satisfy this; Apple IDs, trusted-vendor HSM-protected device identifiers, and digital passports do satisfy this.

If you have a solution that only presents barriers to excessive use and allows abusive traffic to be revoked without depending on IP address, browser fingerprint, or paid/state credentials, then you can make billions of dollars in twelve months.

Ideas welcome! This has been a problem since bots started scraping RSS feeds and republishing them as SEO blogs, and we still don’t have a solution besides Cloudflare and/or CPU-burning interstitials.

(ps. I do have a solution for this, but it would require physical builds, be mildly unprofitable over time with no growth potential, and incite governments hostility towards privacy-preserving identity systems. A billionaire philanthropist could build it in a year and completely solve this problem. Sigh.)

I actually do not have a problem with digital IDs, as long as my personal identity isn't being shared alongside it. Not to the site operator, not to the government.

This might seem contradictory, but I believe this is technically possible? What I don't think is this is how these solutions actually work currently. Like to basically prove that I am indeed a unique visitor who's a person according to the govt, but wouldn't reveal the person info to the site, and wouldn't reveal the site info to the govt, even if they collude.

Same with the whole +18 goof. I'd actually quite like to try age gated communities, like +-5 years my age. I feel a lot of conflict stems from people coming from a bit too different walks of life sometimes. Could even do high confidence location based gating this way, which could also be cool (as well as the exact opposite of cool, because of course).

Assuming a person can only have a single ID, how would that be enforced without a unique party having a 1-to-1 mapping between person and ID?

The party to hold a gov id - site id mapping would be you. The rest can then be facilitated using zero-knowledge proofs, I believe.

I'm not super well-versed in crypto though, so I confess this is a lot more conjecture than knowledge.

So the user generates the ID for each site he visits? What prevents them from generating arbitrary IDs?

The only way I can imagine this working is:

1. You go to the government and request to have a digital ID generated.

2. The government generates a random number.

3. The government issues a request to an NGO to generate a new cryptographic object based on the random number, and receives back a retrieval number.

4. The government gives you the retrieval number, which you can use to get your digital ID from the NGO.

This way, the government only has the mapping between your identity and a random number, and the NGO only has the mapping between the random number and the generated object, with no possibility to deanonymize it because you don't present any ID to get it. Obviously, there must be no information exchange between the government and the NGO.

> So the user generates the ID for each site he visits? What prevents them from generating arbitrary IDs?

The construction would go basically like this:

pseudonym = VRF(secret_key + site_id)

The expectation is that you would have only one valid secret_key at any time, and it would be unknown to the government. This kind of scheme is called anonymous credential generation in literature I believe. It can be established the secret_key got govt backed, but that's it.

The site_id would be e.g. domain cert public key or similar (domain ownership is a moving target, so just the domain name imo is not sound).

VRF is a verifiable random function. This is the magic ZK part.

Pseudonym is what you present to the site, i.e. the identity you go by.

This way the site can verify that this pseudonym was specifically issued for it (making it site unique), and that it belongs to a govt certified identity (of which there should be only one issued at a time per person). The VRF is deterministic, guaranteeing that it's the same person every time.

Revocation is annoying so I didn't bother thinking that through but should be fairly okay I think?

I believe this is robust to people forging arbitrary IDs, to sites colluding with each other in deanonymization, and colluding with the govt in the same. The only kickers I can think of are secret_key misuse (e.g. via duress) / theft / loss / sharing, and the trust anchor (the govt) being untrustworthy (forging invalid or duplicate identities). Would also need to handle people dying, but that would be pretty much just revocation.

I consider trust anchor issues out of scope. The remainder doesn't sound too bad to try defending for, and I think is also basically out of scope.

Potentially important edit: I'm not accounting for timing side channels here, which might be relevant during revocation or else.

Another: didn't mention but in my humble opinion cryptographically attesting people is unsound. People can't calculate crypto in their head, and can't recall long arbitrary strings of hex. What is appropriate to attest (if anything) is their devices instead. But that's a layer of complication I didn't want to deal with here.

>The expectation is that you would have only one valid secret_key at any time

Why, though? If you're the only one who knows it, nothing prevents you from creating as many identities for the same site as you wish.

It's an interactively generated thing, so the govt can ensure you can only complete it once, while being ignorant of its content. Or at least that's the claim of these protocols (e.g. Camenisch–Lysyanskaya (CL) signatures) afaik. I'm not sure how they work in detail.

How sad a world we'd live in where the government has to bless you in order to be able to use the web.

Why assume that?

Person W is welcome to have thousands of unique IDs if they want to, so long as when site X bans identity Y, that ban is applied to all of Person W’s present and future identities. Whether W has a single Y or a thousand Y makes no difference to me. I suppose some sites will care to restrict participation to a single Y per W, but e.g. in the general browsing a site with crawler/bot/AI shielding such as Anubis today, it’s completely irrelevant to them what your Y is so long as rate limits and bans apply to all Y of W rather than to the presented Y alone.

Uhh... Sure, I suppose. I don't think it'd be possible to have that without also making it obvious that two different IDs are controlled by the same person, but sure.

The core of the identity problem is, “how can someone ban you personally without being able to identify you?”, and so far as I know, “a trusted third-party checks your identity and issues you an identifier” is the best we’ve got - but of course those identifiers can be enriched, so you end up needing a single centralized third-party that can issue identities and also honor bans for “the individual behind identity X” from specific sites by site request, while being audit-proven to actually enforce those bans upon me regardless of how many anonymous identities I choose to generate and use. (If you don’t run this as a monopoly, either they all pool their user-operator banned identities lists to prevent ban evasion, which will eventually leak, or they will be compelled to someday later down the road when they lose the inevitable monopoly lawsuits.)

It’s not difficult to solve this problem — the database schema and queries are dead simple! — it’s just exceedingly difficult to succeed if you're not a passport-issuing entity or an authorized monopoly of such.

Specifically, only something government sized is trustworthy enough to not plainly sell your data later... Or perhaps get it stolen.

Scratch that, this happens all the time. With a third party there's no way to revoke, government you can usually physically handle this.

I wrestle with this by just separating this concern.

In the model I described, the trust anchor would be the govt, so basically a centralized model like domain certs. This resolves the issues you list off, but brings others: what if the trust anchor isn't trustworthy and starts forging identities?

The alternative to that would then be web of trust stuff. But this is why I consider this to be a separate problem. If the core protocol could be laid out and standardized at least, then layering on another that makes this choice between centralized vs web of trust could be done separately.

https://www.w3.org/TR/vc-overview/

> We have nothing to protect sites against scrapers except to make it more expensive for everyone’s (...)

Authentication works, doesn't it?

Paid credentials work, yes.

Free credentials temporarily do, but only until someone teachers crawlers to AI-automate signups. Forum spammers already figured this out a long time ago.

A while back someone from Financial Times commented on HN about how it was confusing that we were all so hostile to their free registration required article paywall. The HN community view on their registration requirement suggests that free authentication does not work and has not for some time; has that viewpoint changed in recent years?

> Free credentials temporarily do, but only until someone teachers crawlers to AI-automate signups. Forum spammers already figured this out a long time ago.

Forum spamming is a solved problem, isn't it? I mean, do you see any spam here on HN?

> The HN community view on their registration requirement suggests that free authentication does not work and has not for some time; has that viewpoint changed in recent years?

I don't think there is such a thing as a "HN community view". Are there meetings to agree on a consensus on important topics? Or were you trying to pull an appeal to authority?

> Forum spamming is a solved problem, isn't it? I mean, do you see any spam here on HN?

Yes, I’ve reported two or three this past week or so by emailing the mods. Do you browse /new often? What do you think makes HN a spam-free place?

> I don't think there is such a thing as a "HN community view".

Noted.

> so long as the cost is appropriately small.

there are different metrics for cost, however. Based on cpu utilization and/or time, it's hard to argue that Anubis is a high price.

But if it is important to you to not run javascript for whatever reason, the price of access to a site using Anubis is rather high.

No, no, no, hell fucking no!!

You put stuff on the public Internet, expect it to be read by everyone.

Don't like that? Put it behind a login.

How did the propaganda persuade people into accepting mass surveillance and normalising the invasion of privacy for something that was never really a problem?

I suspect the ones pushing this are the ones running these "crawlers" themselves, much like Cloudflare hosts providers of DDoS services.

What a coincidence that "identity verification" became a hot topic recently.

Don't even try to solve this nonexistent "problem", because that will only drive us further into authoritarian technocracy.

> What a coincidence that "identity verification" became a hot topic recently.

Crying “Conspiracy” in reply to a career Chicken Little is comedic. I’ve been raising warnings about identity verification looming on the horizon for perhaps fifteen years now; thanks to DejaNews for that early realization, I suppose.

> Don't even try to solve this nonexistent "problem", because that will only drive us further into authoritarian technocracy.

I would celebrate and tell all my friends if someone on this thread, on any thread, would explain how we solve this without bankrupting non-business site operators and without a third-party authority. Anubis is a band-aid at best, yet no better solution — not even an idea — is presented alongside your objections.

> You put stuff on the public Internet, expect it to be read by everyone.

My hobbyist forum can barely stay online eight hours a day due to crawler traffic. Someone scraped the entire site by spawning one request per page with no fork limit last year. It was down for a solid week after that, and now has very severe limits in place. I don’t know how they can afford to stay running, but certainly “static only” isn’t going to solve the CPU and bandwidth costs incurred by incompetent and redundant AI crawlers. So, by making their site public in today’s infested internet, their content is no longer accessible.

> Don't like that? Put it behind a login.

As I noted above, one solution is payment — since free credentials registration is not an obstacle to AI bots, after all. For some reason people don’t like to charge money for hobbyist content if they can avoid it. I recognize why and am trying my best to discover a non-monetary solution on their behalf.

> I suspect the ones pushing this are the ones running these "crawlers" themselves, much like Cloudflare hosts providers of DDoS services.

I do not, have not, and will not run crawlers or AI agents, trainers, or other such shit at any time in the past thirty years and will continue to abstain from the entire category, which should be quite easy as I’m a retired sysop now attending full-time accounting school and giving a finger to the entire industry to pursue work that benefits humanity. Same reason I bother telling HN “the anonymity sky is falling” every so often: I’d much prefer it if we didn’t have to sacrifice anonymity online to defeat scraper bots.

> No, no, no, hell fucking no!!

Please find a way to turn your vehemence and passion into a productive contribution, before it’s too late for all of us. As presented, your argument is neither supported nor persuasive, and your hostility only gives opponents of anonymity more arrows in their quiver to shoot at us.

We very definitely do have stuff to protect sites that don't make it more expensive for everyone! Just none of it is open source.

And, fundamentally it can't be opensource. Bot detection (like anti-fraud more generally) is an adversarial game that relies on hidden techniques. Open-sourcing it means you lose that advantage and make life much easier for anyone trying to get around it.

I think there's probably a platform for it that you can open source --- the virtual machine, or the core of the virtual machine or something, but yeah, you're right, this is something Anubis will have to contend with long term; the effective solutions for this all benefit from obscurity.

There's zero reason it cannot be opensource. Proof-of-XXXXX schemes do not rely on obscurity to be functional.

The schemes large players use to increase the cost of e.g. creating new accounts on their services do in fact rely on obscurity. They target developer cost, not compute cost.

But that's not what Anubis (the subject of TFA, and most of the comment threads here) is/are about.

I think it's exactly what Anubis is about? I'm pretty familiar with what Xe is doing.

...You're computing a hash, not making accounts or anything else that relies on obscurity though?

You're computing hash in order to block automated web requests. The hash isn't the point --- the hash cost is the part of this system that isn't going to work long term.

It makes it more expensive for the people running scrapers, and thus not worth it for them in 99.9% of cases, that's the whole point of PoW.

SQRL solves at least part of the ID problem.

Reply on news.ycombinator.com