Hacker News Reader

Comment by ableal | Hacker News Reader

I downloaded the first couple dozen megabytes. It's all nulls.

    $ od 500megabytefile.tar.gz 
    0000000 000000 000000 000000 000000 000000 000000 000000 000000
    *
    27200000

(besides not being a .tar.gz, it's not 500 MB either, it's 512 000 000 bytes = 512 MB, or 488 MiB)

View on HN

This also means that opening the tar.gz does not work. Not knowing about the cheat, I'd want to try. My 7-zip choking on the file would make me download again. After the second time, my trust in Singlehop's transfer abilities would be very low - after all, files they send appear to get corrupted.

So it can basically compress to something extreme, too.

Exactly. I watched the total traffic tick up in Activity Monitor while downloading it, and it was only 2.2 MB in total.

It is sort of a funny cheat though. A good prank to pull on friends to claim you have a very fast internet connection, but for a company to do this in order to promote their product, it is very dishonest.

That isn't strange though -- if I were creating a file that I knew would only be used for speed testing, I would make it al nulls too.

It is just much easier to find a file that is some exact distance if you create it first.

I'd use random data to make it hard to compress. You don't know if there won't be some link layer compression along the way.

That doesn't really fix the problem, what if that random data happened to compress well? The solution is to not compress anything while downloading.

A large quantity of random data is incredibly unlikely to compress well.

"Random data' [1] will not compress well [2].

[1] high-entropy in any of the usually-available ways

[2] for any compression system actually deployed, as opposed to one contrived to match specifically the 'random' source

$ dd if=/dev/random of=nonbullshittestfile bs=1M count=500

That would take forever, at least on my system, as I don't really have a lot of entropy available. Just for kicks I have started this when I began writing this reply. And now I have the huge amount of 200 Byte in this test file. So thats about 10 bytes per second.

But if you take /dev/urandom you can get quite a bit more. On this system it is about 7Mbyte/s.

What's the difference between /dev/random and /dev/urandom? I've only ever used urandom and it spits out data as fast as I can consume it.

/dev/random will block while waiting to collect entropy from the system. /dev/urandom will be satisfied with pseudorandom numbers. That's fine for many applications (e.g. a file filled with garbage) but not acceptable for things like cryptography.

That entropy, by the way, is derived from the keyboard and mouse devices. If you want /dev/random to go faster, move your mouse and type more ..

Can it be configured to use other sources?

Does this mean that if I've got a server with no mouse/keyboard attached, /dev/random will block forever?

Logging into my slicehost server, and running cat /dev/random | hexdump -C seems to support this, more or less - only a few lines get output unless I start typing into the terminal - then it goes marginally faster.

It will also use other interrupt timings in the creation of the entropy, most notably hard drives since they're rather random on when they reply back due to it having to rotate to the right place. Not sure how this works on an SSD. I also believe if you've got some kind of hardware RNG it will use that too.

You can buy hardware random number generators.

http://www.entropykey.co.uk/

http://www.idquantique.com/true-random-number-generator/prod...

http://www.gamesbyemail.com/dicegenerator

...among many others.

Interesting. I don't see /dev/random blocking on Mac OS X (13" MacBook Pro). I wonder what the source is?

On OS X, /dev/random and /dev/urandom are the same thing (both acting like the traditional /dev/urandom).

/dev/random uses environmental noise for entropy, and can be depleted rather quickly.

/dev/urandom supplements its entropy with a PRNG so that it never blocks..

Surely you wouldn't get as greedy as they did though? Nobody would've suspected them if they had've been a little less "cheaty".

I'm guessing it may be even smaller than a file filled with zeroes. If I were doing it, I'd use a sparse file, so it only takes a few blocks, but (sans compression) would take a long time to download. Like this:

  tsuraan@macbeth ~ $ dd if=/dev/zero of=abigfile bs=1M count=1 seek=1000000
  1+0 records in
  1+0 records out
  1048576 bytes (1.0 MB) copied, 0.00320282 s, 327 MB/s
  tsuraan@macbeth ~ $ ls -lh abigfile 
  -rw-r--r-- 1 tsuraan tsuraan 977G Sep 16 11:18 abigfile

I probably wouldn't actually make it ~1TB, but that's the idea.

Reply on news.ycombinator.com