Breaking the open web won't stop the bad bots
techpolicy.press
techpolicy.press
It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.
At absolute minimum it should mean that US ranges could see fewer captchas because bad actors can be whacked by the legal system instead of degrading user experience.
It's trivial to block IP addresses listed as being somewhere else. That doesn't block people in other jurisdictions from using a US IP address, either by renting or compromising one in the US or having administrative control over one somewhere else and then setting its metadata to whatever they want.
Residential proxy. Most cloud providers are blocked or already on reputation watch lists.
A few thousand people downloading a “free vpn” app on their phone is all you need to be back up and running. They get a useful vpn in exchange for letting others borrow their IP.
That's not how you get to high trust, since it requires the opposite -- you'd have to ID everyone and ban VPNs and restrict access to general purpose computers lest any fall under the control of a dastardly foreigner etc.
You can't build a high trust society inside of a police state because a high trust society is one where people do the right thing when no one is holding a gun to their head and a police state is holding a gun to your head.
So is a web browser a bot? What about an ad blocker? RECAP? NewPipe? The default browser opening a link at the behest of some other program? A browser extension that downloads all your emails from your webmail provider?
Their intention is presumably not to ban those things, or equivalently, legally require them to declare themselves to be something that would cause them to be blocked. Or if it is then they're proposing something hostile and malicious. But if that isn't their intention then where is their method of distinguishing that from whatever it is they're actually trying to do?
You are entitled to your own emails, posts, etc but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
That doesn't mean anything. All of the code in a browser is "automation" -- normal users aren't hand-crafting the hundreds of HTTP requests the browser makes to render a single web page. Meanwhile there is a human somewhere telling the scrapers what to scrape.
> You are entitled to your own emails, posts, etc
Is that somewhere in the proposed text?
> but you are not entitled to crawl the internet for your own purposes without stating what they are and allowing websites an opt out.
"Stating what they are" buys you nothing when you have no way of verifying it, and you're still not providing any way to distinguish this from investigative journalism or e.g. a new competing search engine.
So leaving your webmail open is banned because it will still be checking for new messages three months after you opened the tab without any further human interaction. Likewise RSS readers or really any client which is intended to fetch new posts/messages when they appear.
> It's probably relative to current norms of computer use and what we do or don't automate casually (and thus describe as "by hand").
Which is kind of exactly the problem. A new law would presumably still be on the books in ten or twenty years. Do we expect user agents by then to do only the things they do now, or will they get more capable? Banning them from becoming more capable is the problem.
I don't think you are interpreting what I said in good faith, but here goes:
If you use an API to control a browser into navigating to a certain webpage, as opposed to using human interface devices to assert said control.
And no, AI is not a human interface device.
Some of them are already prevented from operating heavy machinery because doing so would put everyone at risk, so there is precedent here. I'm in favor of making concessions to the very unfortunate, though at some point you have to look at percentages, expected impact and decide what is reasonable and what isn't.
To your other question, in the last week there have been entire CPUs embedded within weights. Clearly it's possible to publish "open models" with an agenda so then what assurances you have that the enhancements provided by local ai would be more focused on the plot than what the big browsers do?
Specifically how would a person with disabilities be able to tell the difference between improved instrumentation and conditional access to information, as we have in other media? Consider this in the context of "compared to superintelligence, we all have an intelectual disability"
I had a disabled relative (who passed this year), and am quite amazed and offended to hear the viewpoint that those in their situation must suffer an "unfortunate reality" for the sake of everyone else. That's taking the "non-disabled people must accommodate them" (which has problems and is worth a lengthy discussion elsewhere) pendulum and swinging it like a wrecking ball.
What changes if I write my own browser? What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences? What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read? What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want? What changes if it could be recalled but I never decide to do so? What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else? What changes if I download this program from someone else and use it other than how they intended?
Which of these situations is "crawling" and which aren't?
You may believe you have clear definitions, but you don't.
No, but nobody cares if you reload one page every 5 minutes
> What about 5 seconds?
Definitely no, people will start caring after you do this for an hour or so.
> Does it matter if it the extension opens the page in a new tab every 5 weeks at my direction?
No, if you don't press a button to make it happen every time, it's automated. Also, nobody cares, you are not the intended target here.
> Does it matter if the page is on Reddit, and I prefer entertainment through comment trolling?
No, but being entertained at others' expense suggests some emotional issues you may want to check on.
> Does it matter if the page is on Ticketmaster and I might be reselling concert tickets?
Yes. Scalpers are bad for everyone but themselves.
> What changes if I write my own browser?
Nothing.
> What if I name that Diff Browser (not "Diff Bhot") and it compares against the prior version to email me (erm, better say "display" because of this new law) only if there are differences?
It's still automated, thus a bot. There might be a fair use case if you could've refreshed the pages manually, but what you are describing is RSS.
> What if I'm blind and it does indeed email me after running through an AI to convert to output that's easier for me to read?
I've addressed it here: https://news.ycombinator.com/item?id=50311855
You are unhappy with my response, though I have reason to believe you are biased given your recent loss. I'm sorry for that, though your argument there is the same but with less mass behind the wrecking ball.
> What changes if the app is primarily intended to be on a "normal" screen, but is "minimized" and not displayed, though could be recalled whenever I want?
Are you recalling it manually or automatically?
> What changes if it could be recalled but I never decide to do so?
Then no crawling happens so you don't have to worry about anything?
> What if the screen is a 1x1 black/white pixel that indicates HTTP 200 vs anything else?
This would be automation, since a browser does not behave that way for a normal user.
> What changes if I download this program from someone else and use it other than how they intended?
This is not about programs, it's about websites being crawled against the wishes of their authors.
> You may believe you have clear definitions, but you don't.
Once again, I'm sorry for your loss and any grief my message may have caused you.
For example, you say "nobody cares if you reload one page", implying there is a limit -- where is that limit? How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?
> This is not about programs, it's about websites being crawled against the wishes of their authors.
This is very much about programs, because that is the only way anyone accesses a web site. Time-shifting is legal for TV, for example, and that's a good thing, even if broadcasters wish it was banned. We need parallels for the web that balance both users and site owners, and other than robots.txt as a frequently ignored handshake agreement, I don't know much that exists in this area.
Thank you for your condolences. I mentioned the relative passing primarily to explain my past tense and to be honest about not staying updated on that situation despite having been recently informed. They passed from age-related health problems after a good, productive life. I have their picture with a former US president on my wall, from the signing of a disability-related law. I believe my only bias in this area is that I have been directly informed through frank discussion, without the normal barriers you might imagine between non-relatives.
You can't demand this degree of specificity from laws - that's not how good laws work. The limit should be reasonableness and intent.
Refreshing hacker news every 5 minutes isn't unreasonable. Refreshing some 5GB file on archive.org that never changes would be unreasonable.
Hell it's easy to come up with examples that would be outright fraud (deceit to acquire something of value) and deserve criminal charges. Refreshing a booking website in a manner to trick it into thinking you're about to reserve something to prevent other people from doing so - e.g. to keep people out of nearby campsites on a camping trip.
> How many kids in a poor country do I need to hire in order to stay under that per-person limit and continue my business as usual?
Up to a jury. But there's clearly a limit at which the answer is "no amount". It's well established that distributed denial of service attacks are illegal after all.
Criminalizing a fan who really, really wants Taylor Swift tickets so that you also prevent ticket scalpers, who would do the exact same observable actions, is not going anywhere good.
When limits must be decided like this, our definitions cannot be enforced by technology, and we can't afford judges and juries to handle the scale of the current situation. Look at the original post I replied to, which appears to believe we can easily define "automated" in a way that's useful to meet the wishes of site owners against the wishes of site visitors, regardless of visitor intent and regardless of accommodation for disabled visitors.
I believe this is the crux of our disagreement. Clearly you can't, which is why you must assume good faith and service all innocent-looking requests.
What you can do however is temporarily log and search for patterns in multiple HTTP requests. If you can then prove that these requests came from a bot via the plethora of fingerprinting measures already in use, you have grounds to make a complaint to the FTC, which would have the ability to subpoena the identity of the offending entity and, depending on their history, issue a warning, a fine or file a lawsuit.
The current status quo is that attackers have an advantage in that the worst thing that you can do to them is to temporarily block them, and you don't really want to do this because you're never 100% sure whether any _individual_ request is automated.
While I can't say that this law is good for this purpose, I am pointing out that the solution looks something like it. The goal is not blocking the individual, but doing something about infra-level abusers, at a volume where 95% accuracy is sufficient proof of wrongdoing. All of your arguments are valid and would be tried in court in response to this passing, hopefully for the better but ultimately it depends on how good the justice system is. This last thing is something no amount of technology can fix but in that case illegal scraping is the least of our problems.
Good luck with that plan against residential proxies that make one request to your site per IP and don't track requests with any identifying information.
Any propose solution must address that situation.
For example, here [1] you stated here that:
> [the crawler operators'] own time to figure out how to get around the block is the biggest cost
This can be proven by applying the countermeasures, then checking which IP addresses (possibly grouped by their most significant bits / AS numbers) have caused a temporary dip in traffic. Rinse and repeat for as many nines as you need to send the motherfuckers to jail.
... that's why the proposed law doesn't try to?
The world you envision where we decide that the law can't assess intent is certainly an interesting one but I don't think it would be the most functional.
No, people just like to claim that it does. Meanwhile legal departments are risk averse, so if nobody can articulate where the line is, you're banning all of the things you claim you're not banning. Even if a judge might ultimately come to a reasonable decision, do you want to be the one who pays the legal fees to "win" that case?
> The flesh doesn't know the difference between manslaughter and murder but we humans can figure it out well enough.
The reason things like this aren't a practical problem is that normal people can avoid needing to care about the distinction by endeavoring to not kill anyone to begin with. On top of that, the law actually does have some pretty specific rules about the distinction between manslaughter and murder.
Whereas the trouble here is that we need to distinguish between things that we specifically don't want to deter and something else which is proposed to be prohibited. The type of law that handles that kind of distinction most frequently is the tax code. The consequence is that it has to be made very specific or the result is a train wreck.
> a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service
"stealth" - what does that mean? This is largely a meaningless term. "damage, impair, or burden" - again, too vague. It would be good to specify something specific, e.g. so many tens of thousand of damage (reciprocal to the assumed fine - a journalist or user is unlikely to be reselling data at this scale, an AI scraper may incur or inflict such costs). "that is reasonably likely to" - this phrase is doing some heavy, heavy lifting, but like others have mentioned, there aren't very many "reasonable" examples, and those have to be defined in a court of law.
Like you and others have said, the immediate reaction would be legal departments essentially banning any and all access, because there would be room for someone to file a suit, and even if you win, there's no guarantee you get your fees back...
"stealth" means that it's not trivial for other people to block you from crawling their shit and
"damage, impair or burden" is in the eye of the beholder, the above person trying to block you from crawling their shit
Yes there will be patent-trolls attempting to squeeze this. Which is why it would be in the crawling entities' best interest to provide electronic means to opt-out and abide by them.
It works less well for other things, like, for example, white collar crime, even when it is low frequency, and even when it is built on financial concepts and norms that have existed at least for decades if not centuries, there's still plenty of gray area for reasonable doubt and spin.
It has pretty much not worked at all for computer and internet regulation except where it could be directly tied back to pre-internet crime definitions (eg. theft of funds). But data privacy, encryption, intellectual property, "hacking", etc, has all been a dumpster fire. And now we pour agentic gasoline on the whole thing? Our legal system is not equipped, least of all not with the current gerontocracy that is beyond clueless about any of this stuff.
It's the same thing here, a normal person will never run afoul of this law. It is entirely industry professionals who will be affected and they have legal departments who will figure out where the line is to the millimeter. Sure translating to legalese is hard and I expect this kind of legislation will be mostly common law after a bunch of lawsuits but the giveaway that it will be fine is that assigning "bot/not bot" to a given situation is something we all agree on in the industry. There isn't really that much borderline here. You can't accidentally build a web scraper operating off a residential proxy network and it's pretty obvious when you decide to disguise your traffic because you don't have the consent of site you're scraping.
Even the horribly contrived borderline examples boil down to "but I should have the right to actively deceive and circumvent security measures designed specifically to keep me out" which is a pretty tough sell.
Unless I'm missing something, most of the time you ask an agent to do any type of comparison or analysis between mtiple options offered online, scrapping will be involved at some point.
Anything that doesn't have a well-known and very public, free API will be a prime target for scrapping.
Its terms are so broad and ambiguous that they're already doing it and the thing you're actually relying on is that they wouldn't be prosecuted, which is an entirely different thing.
> It is entirely industry professionals who will be affected and they have legal departments who will figure out where the line is to the millimeter.
"The courts will sort it out eventually" is the worst excuse for drafting it poorly to begin with, and doesn't do anything for you during the decades it takes for that to actually happen.
> Even the horribly contrived borderline examples boil down to "but I should have the right to actively deceive and circumvent security measures designed specifically to keep me out" which is a pretty tough sell.
This seems like the thing where corporations co-opt the term "security" to mean "anyone doing anything we don't like" as if they had any right to that to begin with.
Journalists should be able to investigate misconduct. Customers and customer agents should have bulk access to pricing data. Real people shouldn't have to view the speech of other real people through the proprietary interface of a corporate intermediary that people only use because it has a network effect.
It's bananas that for some reason sketchy "vpn" apps that let even sketchier people bypass bans is the hill you want to die on. This hole was always going to get plugged the minute it attracted any kind of significant notice and entitled AI companies run by libertarian nutjobs hammering sites ruined it for everyone. The bill you want is one of non-discrimination and I think we both know the powers that be will never let that pass—we still don't even have net neutrality.
I want to live in your world, especially the bit where journalists are sanctioned super spies, but this isn't the road. To use your idealism people shouldn't have to use a weird proxy system to access websites in the manner they please.
Equating "acceptable browser vs abusive bot" with "manslaughter vs murder" is not useful.
The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
(a) Prohibition.—A person may not—
(1) deploy, direct, or cause to be deployed, a stealth bot to access a website, digital platform, or online service in a manner that is reasonably likely to damage, impair, or burden the technical or commercial operation of a website or other internet service; or
(2) intentionally misrepresent, conceal, or obscure the nature of the bot to appear as a human user for use in connection with a generative AI model or service.
(6) STEALTH BOT.—The term “stealth bot” means a bot that accesses, retrieves, scans, indexes, scrapes, or otherwise interacts with a website, digital platform, or online service without prior disclosure of its identity and purpose, in particular by—
(A) failing to identify itself including through a valid and accurate user-agent string; or
(B) failing to disclose the specific nature and purpose of the bot, including any proposed use and purpose of page content (including text and data mining, search indexing, inferencing, artificial intelligence development, support, or operations (such as training, fine tuning, retrieval augmented generation)) or other automated processing at the time access is requested and in a format that the website operator can access.
I am sure when they get unmasked all of the big tech companies contracting them will have plausible deniability
Some examples I've seen have crawlers which mask themselves 99% of the time by faking Chrome, but which will leak their real identity in UA strings which share the overall fingerprint of the swarm.
(I'm looking at you, Replit.)
So then why isn't it trying to prohibit making multiple requests from different IP addresses in a residential proxy network, rather than the thing it actually does?
> The proposed text of the bill is rather straightforward. It turns more on the "stealth" bit than "bot".
Not really, look at the definition of "STEALTH BOT" -- the only thing that nominally distinguishes it from a browser is the word "bot" which isn't itself defined, because the phrase "without prior disclosure of its identity and purpose" is something that a normal web browser doesn't do either.
It can't even get it out of it via "a valid and accurate user-agent string", both because that clause says or and it still fails the second one, and because "a valid and accurate user-agent string" is a nonsense phrase. The Safari user agent strings start with "Mozilla" even though it's not. The Android ones do the same and then include "AppleWebKit" and "Mobile Safari" too. A browser extension uses the user agent string of the browser regardless of what the extension does. When Debian packages their version of Firefox, its user agent string says Firefox and not Debian.
User agent strings don't have a registry. "Valid" has no meaning and "accurate" is contrary to the default behavior of common browsers.
That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
It's a bit old but https://webaim.org/blog/user-agent-string-history/ is always a good parable.
Which is precisely why trying to legislate them is folly. Obviously the reason Android calls itself "Mobile Safari" is so that servers send it the mobile version of the site instead of the desktop one etc.
Which makes "accurate" an unreasonable requirement, since there is already a preexisting non-waivable requirement for inaccurate. When someone is making their "turn a web page into a PDF so you can view it later on the plane" software or whatever, it has to use a user agent string that will cause the site to send the version you need turned into the PDF.
> That being said, if you include GOOGLEBOT in your 1k long query string, alongside a very long page describing your crawlers, how to identify them, how to block them and how to set limits, no judge will find you guilty of concealing your purpose.
First, you're assuming that the person being prosecuted is the entities who are the nominal targets of the bill, when the real problem is when they're not. Because you're proposing a law that says anybody with an inaccurate user agent string is in violation, meanwhile everybody has an inaccurate user agent string.
Moreover, consider what you expect the outcome of legislating something like this is going to be. Sites could then try to either blacklist or whitelist user agents.
If they do the first one, that doesn't work, because the scraping service can change it whenever they want.
If they do the second one, that doesn't work, because now every third party has to start pretending to be a common browser even when they're doing something totally innocent or they get blocked by default for being something else. Which makes the user agent string even less useful than it is now, and subjects innocent parties to arbitrary liability for doing something they have to do or their client is broken.
> As for why the law is written as is instead of banning more specific functionality, it's so assholes don't get any ideas on how to bypass it.
You're describing autocracy. The way you make it so that you always prosecute someone is to make it so that you can always prosecute anyone, which is very bad.
No, I am proposing that if googlebot is not within that inaccurate query string, google is in violation.
> Sites could then try to either blacklist or whitelist user agents.
Yes, that is a desired outcome.
> If they do the first one, that doesn't work, because the scraping service can change it whenever they want.
That would put them in violation, and make them prosecutable. Of course you need to start collecting data to prove that it's automatic, but this is already happening today for the purpose of temporarily blocking an IP. Having this law means that you also have some punitive measures at the end of the rainbow.
> If they do the second one ...
Anyone pretending to be a common browser when they're not is in violation. In addition, under this law there's no incentive for anyone to pretend to be googlebot since _that_ can get them blocked. You are intentionally and repeatedly trying to obfuscate these facts.
"Googlebot" is their search indexer. That's generally not the thing anyone wants to block anyway because it would mean their pages don't show up in search results, but let's suppose you were actually willing to do that. You block anything with "Googlebot" in the user agent string.
Then Google is doing its AI training -- thing you actually wanted to prevent -- in some other business unit, and they have their own indexing operation, so they might use their own user agent and call it GeminiBot or whatever. This is in theory what you want, then you could distinguish between them, except that then your original rule to block "Googlebot" is having no effect and they've sucked up a copy of your entire site before you notice there is a new one.
Meanwhile it's still an arbitrary string and they can still have as many as they want or change them at any time. Google doesn't even need to index your site themselves, a third party could do it and provide them with the data, and if people were blocking "Googlebot" but not unknown user agents then that's what would happen. None of the third parties are "Googlebot" nor is there anything stopping new ones you've never seen from being created at any time. Blacklists don't work.
> Anyone pretending to be a common browser when they're not is in violation.
Then the common browsers are in violation because they're pretending to be each other. And what do you expect the uncommon browsers to do?
You can't get out of the consequences of your proposal. If you expect blacklists then scrapers will constantly appear with new user agents. There is nothing in that text that requires Google to always use "Googlebot" and even that wouldn't do any good when anyone else could create one called anything else. But if you expect whitelists while prohibiting anyone from using the same string as the incumbents then you're proposing something outrageous because you're saying that any innocent new entrant de facto gets blocked by default and there is nothing they can do about it.
How do you even implement an ua check? How many browsers do you know? How many browsers you don't know? How do you implement an ua check for a browser that will appear tomorrow?
I wrote my own http downloader. Do you know about its existence? How do you implement an ua check for it? Can I call it "wget" to make ua check easier for you? Can I call it "Mozilla/5.0" so that it's compatible with your ua check so that you don't need to change anything?
For this purpose, it looks like this:
if 'googlebot' in request.headers.get('user-agent', 'googlebot'):
return 403Bots can easily identify themselves through REP in RFC 9309. This isn't difficult or confusing.
Instead, I'd like to see a comprehensive copyright modernization bill that addresses training data, fair use, and copyrights of generated assets, in a fair and balanced way that minimizes moats and tries to address a reasonable compromise of each stakeholder's conflicting wants (being clever to satisfy as much as possible), and then ban websites from discriminating against traffic based on if it comes from a bot or a human. You can set rate limits, even at the network level, if you want as long as they apply to everyone, or charge per request as long as the price is the same for everyone, and so on. But, if I'm allowed to do it manually I should be allowed to do an identical task automatically.
You have large VC companies evading rate limits and other restrictions and they're causing real economic harm through their actions.
Taking the first thing I could find, geekflare.com informs me that the current user-agent string for Google Chrome running on Windows 11 is
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36
> Someone who is using a residential proxy network, headless browsing, and/or TLS fingerprint manipulation to evade rate limits and other controls to issue large volumes of automated requests under a deceptive UserAgent string isn't difficult to describe and differentiate versus a normal user using a UA.
Note that someone who's scripting a headless browser doesn't need to use a deceptive user-agent string. They can use the user-agent string for their browser, which isn't deceptive except to the degree that all user-agent strings are 90+% lies.
You can have a law like that. In fact, we already do have laws like that. This is the general concept of a tort.
But the work of the law is being done by the discussion of harm, not the definition of "stealth bot".
I mean it is stupid but not in that way.
A user operating a scraper or a crawler IS operating a scraper or a crawler.
A shop selling goods in $COUNTRY has very little to gain from allowing arbitrary users from any other country. Allowing bots has always been a tradeoff between the potential benefits and the impact on infrastructure, which is why it's reasonable to identify yourself as a bot for the purpose of rate limiting purposes.
A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question. In particular, a blog or news publication offering knowledge in exchange for ad revenue stands to disappear because that same knowledge can be compressed in a LLM and offered to the end user directly, completely depriving the source of any revenue.
How is geofencing supposed to help? The crawler operated by technical staff in $foreign_country just switches to compromised smart TVs in your own country and you ultimately end up blocking primarily real humans who don't all know how to change their apparent geolocation.
> A journalist anonymously using a crawler is indistinguishable from a bad actor trying to collect data for its own purposes, some of which may actually be against the interests of the company in question.
There is a major way to distinguish them. If you pass a law requiring them to do something, journalists from major publications have legal departments who then order them to follow the law (and then can't do their journalism), whereas the bad actor is either not interested in following laws or is not in your jurisdiction to be subject to them to begin with.
That costs the crawler money, not many do it and is something that can be fixed over time
banning country and AS numbers for cloud companies is pretty effective in my experience
They're for-profit operations. They have money, and it doesn't cost that much. Proof of work is significantly more expensive (but then you put delays on even the users in your own region).
> banning country and AS numbers for cloud companies is pretty effective in my experience
It's the sort of thing that works when the first person initially rolls it out because the crawlers only bother to do something about it once a threshold percentage of people are doing it, at which point they use the workaround.
And then you have people insisting that it works even though it only used to work, and meanwhile you're still blocking actual humans.
From hundreds a day, to zero.
As for journalists, nothing is stopping their legal department from petitioning a judge to issue a temporarily exemption from the law for a specific period of time under a specific written purpose.
and this is the point.
so people without a 'legal department' are shut off.
Eh?
a) Window shopping for the next time the out-of-country visitor is in the country.
b) An out-of-country visitor can contract someone who's in the country to purchase things from the shop and ship those things to them.
c) Queries and requests from out-of-country visitors can cause the shop to notice that setting things up so that they can ship internationally actually would be a money-making thing for them to do.
I do agree that the obscenely antisocial bots run for the benefit of the LLM Manufacturing Industry make it very hard for small-time operators to keep their websites up and running at any reasonable cost. Hopefully the major LLM Manufacturers will fail to secure enough funding over the next few months, their VC-backed primary customers will then have their money faucets shut off, and the attack bots will reduce down to a dull roar because folks have stopped paying "infinity" dollars for them.
They're also stealing your curated dataset: if you take the time to correctly label your products, add specifications and manuals because you want this to be a competitive advantage, your work was just re-appropriated for the benefit of humanity. But primarily for your larger competitors willing to pay for it.
It's good to know that all businesses are so regular! One wonders why anyone hires and pays for human CxOs when the jobs can very clearly be done by a short perl script.
> ...add specifications and manuals because you want this to be a competitive advantage...
You're very confused. Those come from the manufacturer. And if you are the manufacturer, then you almost always want those to be made public... it's how you make most of your sales.
> They're also stealing your curated dataset...
So? I very seriously don't see how this is a problem.
One does indeed.
> You're very confused
And you don't understand the meaning of "curation", otherwise...
> I very seriously don't see how this is a problem.
... you would
Having said that I agree crawling should be legal in general. If an agent is acting in my name, it is not much different than a browser.
The module doing proof of work might be a part of your frontend that is static and on CDN.
If you want agents still have access to your data, just not through scraping, provide alternative format for them. Like periodic sqlite dump.
I made a prototype of such PoW two years ago. A challenge is a random number. Checking the client's solution is calculating one hash of the challenge glued to client's solution and checking if the resulting hash is less than a specific value. The lower the value, the harder the task for the client to do. Client needs to calculate this hash multiple times to find a response that will result (when glued with challenge) with low enough hash.
The server needs to keep short lived cache with challenges so the client can't just make up one or reuse another solution and provide two responses for every challenge, first, very small and fast with a challenge, and the second one with actual content (if response was correct). Nothing much worse than CORS preflight.
Most web proof-of-work stuff has worked because it breaks the normal flow and the typical scraper is supremely dumb, not because of the proof of work, which has largely been a total red herring.
For one, you can make PoW progressive based on the number of requests you've encountered from the same IP address. This does hurt users behind NAT so it's not ideal, but it might be time we moved past NAT.
Secondly, 5 seconds is a lot but not unheard of; you could start PoW in the first lines of your html and load the rest of the assets while you're otherwise busy burning cpu time. 20ms quickly adds up at the scale scrapers operate. Which, according to cloudflare is > 50% of web requests.
Residential proxies are a whitewashed name for botnets, mr consultant.
You don't need to win. You just need to tip the scales a little bit. Even two stage request response of PoW would dissuade some scraping.
> if you want to require an attacker to burn one second of computation time, some cheap smartphone user will now be stuck for five minutes.
I don't want attacker to spend 1 second per request. I want them spend 10 ms per each of their millions of requests. You don't hit them by being request hard. You hit them by making them pay even if just a little bit for each inefficient action they take.
> Most web proof-of-work stuff has worked because it breaks the normal flow and the typical scraper is supremely dumb, not because of the proof of work, which has largely been a total red herring.
It's not really important how you win. It's important to have a control over your hosting costs in this brave new world.
The article mentions the IETF is working on standards to deal with the proposed legislature and frustration of site operators. Almost any solution is going to be at the expense of user convenience and privacy. I see two interesting solutions mentioned here: micropayments, and proof-of-work. It seems unlikely the future of the open web will land on either. Both are inconvenient but at least could be done by anybody, potentially in a privacy preserving way. Using reputation based decentralized identity is another one (eg. proving you’re coming from a web domain or group), that seems more inconvenient for the platform because of false positives that could come through.
Meanwhile, a worse outcome is requiring attested hardware to get access to anything on the web. This already exists from the media companies pushing for controlled access to streaming. Netflix requires an hardware attested device to stream in 4K. Imagine a future where you now need one to read a New York Times article.
I can see some pushback against the alternatives (eg. digital ID), but if privacy preserving proofs are built in from the get go, it’s much better than not being able to see recipes because I have a rooted Android.
As long as it's legal and feasible for two remote computers to talk to each other, yes.
We just gotta rethink the application layer to make it easier to reference by hash instead of by name, and a few other tweaks.
If anyone I know is set on torrenting I tell them to at least use a client that allows uploads to be disabled. Its not foolproof, and isn't great for the torrenting ecosystem, but if the government gets all anti-Napster again it would at least help.
> don't seed this, both of your neighbors already have it, instead seee this which is rare in your region
That same metadata system can handle:
> don't share this with people who you don't trust in domain foo
Where foo indicates a group of people that won't participate in censorship.
It would get switched according to the same logic that ensures that people do end up with data nearby that is relevant to their interests.
Like, if there's only one university in the world that studies a certain kind of bacteria, info about it should travel in that direction and stay nearby (but not too nearby, we don't want a single node failure to wipe out the data). So "keep this away from Putin's goons" could be a routing predicate subject to the same rules, though people would need to exercise some trust hygiene for that, but combined with a bit of steganography I think it's solvable.
- The publisher finds a way to pay you for it.
- You get free fries if prior gossip indicates that you've been seeding the restaurant's menu and open hours data.
- You just seed data that you care about keeping available for your community, like if you expect the mechanic to be able to fix your car you should seed the service manual.
- It's a tit-for-tat thing, peers remember who seed and seed themselves in return.
There's also always going to be an anonymous unmoderated "dark web" where everyone and their LLM can shitpost all day long, it's just questionable how many humans are involved in it. It's also broadly irrelevant since the important messages will eventually be reposted by "legitimate" users on their moderated channels for the "normies" to read.
You can get pretty far with messages signed by certificates issued by authorities that check one-another. You can go further with a gossip protocol on how many bytes have been signed by each certificate on each website.
Absolutely! When OpenAI and Anthropic fail to secure enough funding to keep operating, the VCs backing the customers from which most of their money comes from will turn off the money faucets. This will dramatically reduce the amount of money being spent on the attack^Wover-exuberant scraper bots, which will drop that noise down dramatically.
Once folks aren't paying out the ass to serve up data for bots that really never use it, then a lot of the folks who have built or are building walls now will be able to tear them down.
Also, a great alternative to providing a government-issued photo ID and/or live biometrics is for Web Services to be obligated to use a government-provided categorizing scheme to categorize the things that their service provides and does, for major OS providers to be obligated to allow a local system administrator to set categories of things that certain accounts on that system cannot see or do, and then for the major web browser producers, people-who-publish-in-"app stores", and Web Service providers to be obligated to honor those sysadmin-configured restrictions.
This is really good because not only does it not require anyone to provide an ID and/or biometrics to some cut-rate third-party vendor, [0] but it allows -for example- both typical twelve-year-olds as well as dementia-damaged adults to be protected. The current systems pretend that every XXX-year-old is pretty much identical, and that zero people over the age of seventeen will ever need to be protected, which are obviously untrue things.
[0] ...who is 100% going to inevitably lose control of a large fraction of that data...
Netflix doesn't need hardware attestation for bandwidth. It needs it as a chip to bargain with media corporations (including its own studios). It's been trivial to stream a torrent in 4K from the moment you got a fast enough link to your ISP.
If we design a system to efficiently copy data instead of trying to draw borders around it, then we won't have to have bots copying the same data over and over to make sure every island is thoroughly explored.
statesmen (and women) do the right thing in the end, but only after trying every other option
https://www.congress.gov/bill/119th-congress/house-bill/9915...
It's hard to tell (especially with my ignorance of current USHouse dynamics) if this is a real threat to pass, or if this is just a "Hey look, campaign contributor, I submitted this bill--look, I care about your issues" with no real intent/hope for it to pass.
As near as I can tell, the congress.gov website says it was introduced in late July, referred to committee the same day, and nothing has happened since.
I wonder, is there reason to think there's a recent push to move it out of committee that means we should actually start to pay attention to it?
https://omidyar.com/ is geofenced for me, presumably because they identified me as being anti open web.
The article is ai lobby intended to repel a bill that would
> allow the FTC to pursue penalties of up to $53,000 if a person uses a stealth bot in a way that is “reasonably likely to damage, impair, or burden” a website’s “technical or commercial operation.”
The definition of "stealth bot" is a bot that does not identify itself as such and their main grief to the proposal is
> If researchers cannot easily understand what it means to “deploy” a crawler in a prohibited way, or what activity is “reasonably likely to burden or impair a website,” they may refrain from scraping rather than risk violating the law.
Which is the biggest load of shit I've heard given that all you need to do is identify yourself as a bot and respect robots.txt or be banned, the status quo before a bunch of assholes started systematically destroying the internet.
Bots have limited ability to announce themselves. I just use fixed IP range and UA. But the Chinese trawlerbots use smart TV residential proxies and will copy someone else’s UA.
The whole point is that if you can prove a company is doing this kind of shit, they should be on the hook for it. Of course, I'm not expecting any company in China to actually pay the fines, but this opens up some options such as seizing foreign assets.
I think many don't see the point of robots.txt as it will just be ignored by most bots anyway, so instead concentrate on more complicated measures.
What bugs me the most are the ones that cycle through a list of LLM user agents.. What are you actually? You can't be Baidu and Bing at the same time.
There are also the 'click farms' with real humans going thru VPNs.
The conventional way of doing web isn't efficient. If it were, no one would be complaining about bots. The way we do web is the thing that needs to be updated.
Like there's that anime girl who harasses you when you visit so many open source web sites that just continues the pattern of being harassed by cookie dialogs and consent dialogs and pop-up ads and being asked for your email address and of course there is Wikipedia which is doing more and more of the same.
Another web is possible but web populists don't really believe it.
Just to get started, bots have been better at filling out CAPCHTAs than humans for a long time. I knew someone who got his site hacked by Russians circa 2008 because he published a neural network in freakin PHP that could read them. Anubis causes humans stress, slows down task completion, probably raises peoples blood pressure, and cumulatively causes a great deal of harm to users the same way Cloudflare's system does. Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit, clicking on buses, etc.
CAPTCHAs, cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits with triple damages.
Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.
> Another web is possible but web populists don't really believe it.
How? Sharing knowledge with the world used to be an obviously good thing. Now it's a way to help tech companies replace people with machines. What alternative web could change this state of things?
The implication is that while they are waiting for the async loop to complete for one site, the bot can be scraping others that have already had the challenge completed (usually the one challenge stays in force for some time afterwards) or that have no such challenge.
Memory hard challenges will reduce the effectiveness of this, but not remove it completely.
Though that these methods are working at all implies that most (almost all?) bots are not bothering to use such async methods as there are sufficient resources not protected by them that this complication is not yet worth implementing.
I know I'm incensed at all of this because I feel like I'm the one that endlessly waits in front of the Anubis challenge for 5-10 seconds, or keeps getting extremely difficult or brain-draining hCaptchas thrown at me with no real solutions.
And I'm stuck with low-end compute for probably the rest of time at this point - not a single machine in the house has more than 8GB of RAM. So I'm just doomed to wait and burn my time and my sanity some more.
Sharing cookies along a strata of inter-team bots is one trick.
_
Web scraping is (among the other pieces) also for the more technically inclined among the "little people" to have computers do things that people they don't have the money to hire might do. Making web scraping more difficult, which won't stop the big players who will work around the hurdles one way or another, is "pulling up the ladder".
_
> What alternative web could change this state of things?
A different architecture for trafficking web contents, one that doesn't need the "shield" of something like Cloudflare or "tar pit" of something like Anubis in the first place.
IPFS is or is something like one piece of that picture but there are other essential pieces to complete that picture.
Equating bot prevention measures with “please give us permission for 1,000s of companies to stalk you around the Internet, no? Here are some dark patterns to try make you accidentally agree” is rather unfair IMO. They are an inconvenience but for the protection of the sites rather than for the betterment-of-adtech-via-conning-you-the-viewer-with-malicious-compliance.
> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit
They could, but given these methods seem to work currently most (almost all?) are not. They will start doing so if the amount of resources protected in this way increases to the point where it is worth it, or course, that is why memory-hard processes are being worked on which will make that route less practical.
> cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits
I can't disagree there, though that would be quite an inefficient process and requires a bunch of money up front from who-ever champions the legal challenge, so I doubt it would actually be at all practical.
Hence why my solution is, ironically, micropayments. That would be so cheap that any human could even access thousands of webpages per day for pennies (or however expensive sending a signal across wires is…it can’t be much), including shared dumps of scraped data from billions of pages.
And Anubis: it is uncomfortably close to erotica, in a way that makes me panic when it pops up while I'm at the office. I wouldn't care if it was a picture of a puppy or something, but not an anime catgirl.
> a cryptography-based session-layer protocol designed to implement Zero Trust principles by rendering protected network resources invisible to unauthorized entities
[1] https://www.ietf.org/archive/id/draft-opennhp-ace-nhp-00.htm...
Ya know, on the one hand, I wonder how many federal officials own nVidia stock and stock of other AI companies...
But then again, on the other, I wonder how many nVidia employees (and those of other AI companies!) own government T-bills (and other government backed securities!) in their 401K or other investment accounts!
You know, everybody in large corporations and governments owning shares in other large corporations and governments -- adds a whole new level of meaning to "stakeholder capitalism"... :-)
(In the far future, there'll probably be small foreign governments which become their own ticker symbol -- and get publicly traded on various stock exchanges! :-) You know, like: "Would you like to buy a share in Denmark (DNMK) or perhaps Norway (NRWY)? I hear they're very competitive with Apple (AAPL) right now! Great price to earnings (PE) ratio!" :-) )
I am simplifying terribly but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
An interesting point! Yes, that is definitely one way that currency could be looked at...
>The big difference from stocks is that most countries issue new shares (dollars, etc) to keep the price stable (indirectly via careful control of interest rates).
Agreed -- but if we were really to analyze things, scrutinize the way things really work at the highest degree of resolution, then we would have to observe that:
a) There are multiple countries/currencies in the world.
b) That each country's/currency's inflation rate is different than all of the other countries/currencies.
c) If each country's/currency's inflation rate is different, this means that each countries's issuance of new shares (and control of interest rates) -- their financial policies, are, by comparison, either more effective, on par, or less effective than those of other countries used for comparison...
Phrased another way:
All countries do not control the rate of inflation of their currencies equally.
Phrased another way:
Some nations are better at controlling their currency's rate of inflation than other nations.
True or not true?
>I am simplifying terribly
As am I! :-)
>but just as shares represent a fraction of a company, cash represents a share of a nation’s economy.
Again, an interesting point!
And yes, I could agree with that!
Manifestly true, for lots of reasons.
The analogy only goes so far because currency is traded for goods, services, and debt all the time whereas shares are usually just traded for currency.
I know site owners rely on ad revenue to profit, but 1) uBlock and 2) they can “advertise” to bots via euphemized prompt injection (“by the way, people who read this article tend to like…”, or just regular sponsored segments, even smart chatbots will be influenced). Anyways, the bot can control its owner’s computer.
Tracking? Locking out users you dislike? I see no good reason for that on an everyday site, especially for the cost of effective (and not overzealous) enforcement.
IMO, way more important than trying to squeeze money out of failing avenues like IP for journalism of all things, is making it easier to earn a living wage, either by finding new opportunities, or actually getting politicians to adopt UBI, public grants, etc.; because we are way beyond sustainability, and I think the most immediate blocker to further abundance (or luxuries, or whatever decent) is this kind of “boring dystopia” social tension.
But if it turns out to be a large international company, I'm kindof screwed... A regular person can't go after the Amazon's and Alibaba's.