Hacker News Reader

Comment by judge2020 | Hacker News Reader

judge2020

> What's most interesting is that the bucket is private, so the only way they could identify that there is something malicious at a URL is if someone downloads it using Chrome. I'm assuming they make this decision based on some database of checksums.

Doesn't Chrome upload everything downloaded to VirusTotal (a Google product)?

View on HN

> Doesn't Chrome upload everything downloaded to VirusTotal (a Google product)?

It doesn't, unless you opt for SafeSearch "Enhanced Protection" or enable "Help improve security on the web for everyone" in "Standard Protection". Both are off by default, IIRC. Without it, it periodically downloads what amounts to bloom filter of "potentially unsafe" URLs/domains.

On the other hand, GMail and GDrive do run the checks via VirusTotal, as far as we know - which means that OP case may have been caused by having some of the recipients having their incoming mail automatically scanned. It's similar for Microsoft version (FOPE users provide input for Defender Smart Screen), at least last time I checked.

What happens if it is a hit against the bloom filter / checksum? Would it transmit the URL so that it can be blocklisted?

https://developers.google.com/safe-browsing/v4/update-api#ch...

TL;DR is you download a chunk of SHA-256 hashes and check if the hash for your URL is there. There is of course the chance of collision but that is minuscule.

Oh I know that's how that works, I meant, does Google transmit back the URLs once it does get a hit, to protect others from downloading that file?

Why would it need to do that? To protect others from the same url, the same hash checking method should work.

The blacklisted URL in this case is found in a downloaded file from a S3 bucket.

Other people downloading the same file would get the same "protection", but in this case this goes a step further:

The S3 bucket itself gets then blacklisted. As it was a private bucket, one of the ways this could happen is that once chrome found the blacklisted URL, it sent back to Google the url (s3 bucket) where the file with the blacklisted URL was found.

The hashes of all things that match a "probably evil" bloom filter, yes.

Hosting a virus on a domain and then downloading it a few times with different chrome installations sounds like a good way to get the whole domain blacklisted...

That's why user uploads are worth some thought and consideration. File uploads normally gets treated as a nuisance by developers because it can become kind of fiddly even when it works and you are getting file upload bugs from support.

It normally isn't that much of a challenge to mitigate the issues, but other things get priorities. Companies end up leaving pivots to XSS attacks and similar bugs too.

Google has a great service for this called Checksum. You upload a file checksum and it validates it against the database of all known bad checksums that might flag your website as unsafe. The pricing is pretty reasonable too and you can proxy file uploads through their service directly.

I'm actually not telling the truth but at what point did you realize that? And what would be the implications if Google actually did release a service like this? It feels a bit like racketeering.

Real shame if this domain got blocked because of a contraband file, eh? Just pay us and we'll make sure you don't have any problems.

Ha! You got me. I was like, wow, that sounds really useful. I'd love to sign up for that, and built my app to use it, if that were the case.

But then, I realized: 1). I'd be integrating further into Google because of a problem they created (racketeering), and 2). They seem to really dislike having paying customers (even if they made it, they'd kill it before long).

And 3), they would later update their evil-bloom-filter and all of the sudden the file you paid to get verified is now an Evil File, and they blacklist you anyway.

They actually blacklist you even faster, because of course they have in their database that you have the now-evil-file.

I wonder if that could be triggered even when the certificate chain is not validated... you could MITM yourself (for example, using Fiddler) and make Chrome think it's downloading files from the real origin. In that case, an attacker could do that from multiple IPs and force Google to flag your whole domain.

Why isn't Dropbox blacklisted? Too big?

Dropbox actually provides an unique domain for each and every user - and separates the UGC from the web front code and Dropbox own assets that way - that's where the files you preview/download are actually coming from. I have no doubt a fair number of those is blacklisted.

unique TLD? that should be very costly?

or does GSB not ban the entire TLD when a subdomain has malicious content?

Would be great if our overlords at least publish the overzealous rules we need to abide by.

Dropbox DL and Preview urls take a form of https://uc[26 character hex string].dl.dropboxusercontent.com/... and https://uc[26 character hex string].preview.dropboxusercontent.com/... - it does not have to be a separate TLD to avoid being blocked, but it has to be differentiated.

This is the same reason why the block of the TFA company did not cause an outage of everyone using CloudFront - GSB does not block full TLDs if it can be shown content is distinct. Same for anyone using S3, Azure equivalents and so on.

I wonder if there's a threshold here.. When I researched this issue (while we were figuring out how to mitigate it), I did encounter some people who had their entire domains blocked because 1 subdomain had bad content. In fact, this thread itself has mention of that happening to neocities

Author here. It's really not clear what criteria GSB uses to decide at which level the ban should apply.

Probably when the ratio of bad sites to good sites at a particular subdomain level passes a threshold.

Or when a significant litigation risk is perceived, if the domain level block review is human

My Google-fu is failing me right now, but there is a list of domains like dropboxusercontent.com that are treated as pseudo second-level domains for purposes like this.

e.g. u1234.dropboxusercontent.com is treated as a unique domain just like u1234-dropboxusercontent.com would be.

Edit: here we go, from another comment - the Public Suffix List: https://publicsuffix.org/

Sounds rather too resource-intensive? I've just tried with current Chrome on Windows and a 32MB zip on my personal domain, Wireshark says the file has not been sent anywhere.

I believe there are limits on the virus checking size. You can see this when trying to download really large files from Google drive (> 100mb)

https://developers.virustotal.com/v3.0/reference#files-scan

Seems I might have just hit the limit? ... Nope, 8.1MB zip file also wasn't sent anywhere.

Wouldnt it be more efficient to grab it in parallel to your download?

That only shifts the bandwidth cost between the original server and the user, Google's resources are unaffected. And it's not what GP claimed.

I just checked the nginx access logs - both the 32MB and the 8MB zip files have been accessed only once (both were created only for this experiment).

Reply on news.ycombinator.com