Hacker News Reader

Hacker News

Comment by ranit | Hacker News Reader

Parent Full thread

ranit

What did you use to sort these 10 million lines?

View on HN

sort in bash.

Specific command:

LC_ALL=c sort -n filename -r > output

but what did you use to sort by the second field, in your original setup?

LC_ALL=C sort --field-separator=',' -k2 -n

This uses field 2 to the end of the line as the sort field; would

  k2,2

have been faster, as it limits to just the second field?

There were only 2 fields so I expect not.

This is really interesting to me - I can't seem to reproduce this on a couple different platforms. In the past (and just now) I've definitely seen 100x differences with LC_ALL=C, but you used that on both, I'm sure.

How large was the 10 million line file, and do you know whether sort was paging files to disk?

About 300MB. I just tried to replicate using random data and failed, will look more later, mine is power-law distributed and might make a difference.

Used these commands:

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 10000000 > text

cat /dev/urandom | od -DAn |tr -s ' ' '\n' |head -10000000 | sed -e 's/$/\/9.0/'> num

bc -l num > num2

(that's to create decimal numbers)

paste -d, text num2 > textfirst

paste -d, num2 text > numfirst

time LC_ALL=c sort -n numfirst -r > output

real 0m10.712s user 0m21.959s sys 0m1.644s

time LC_ALL=C sort --field-separator=',' -k2 -n -r textfirst > output

real 0m9.638s user 0m21.940s sys 0m1.293s

Reply on news.ycombinator.com