Comment by burntsushi | Hacker News Reader

Comment by burntsushi | Hacker News Reader

Parent Full thread

I assume you're asking why the "simple" Rust version is slower than the "simple" Go version?

I wrote the simple Rust program. I've also been writing Go for a long time. It's tough to say precisely why, but here are some guesses:

* The simple variants, by virtue of being simple, do a lot of extra allocation. Go, because of its GC, might be able to do a bit better here. (I believe the Rust program also does more allocations than the Go program.)

* In Rust, all strings are UTF-8 validated. In Go, they are not. In Go, strings are only conventionally UTF-8.

* A good portion of these programs is spent interacting with a hashmap. Rust's default hashing algorithm is chosen to prevent HashDoS attacks[1] at the expense of slower hashing. I actually don't know whether Go's hashmap does the same. So I'm highlighting it here as a potential difference that perhaps someone else can elaborate on.

[1] - https://doc.rust-lang.org/std/collections/struct.HashMap.htm...

I had a specific question about the rust implementation.

  ordered.sort_by(|&(_, cnt1), &(_, cnt2)| cnt2.cmp(&cnt1));

would produce the same result as what was in the blog post:

  ordered.sort_by(|&(_, cnt1), &(_, cnt2)| cnt1.cmp(&cnt2).reverse());

But would avoiding the `reverse` call make it any faster, or is that a zero cost abstraction?

I didn't check the codegen or anything, but:

1) Yes, almost certainly zero-cost. 2) This isn't a hot part of the program.

See also: https://old.reddit.com/r/rust/comments/m5ix0s/performance_co...

> prevent HashDoS attacks[1] at the expense of slower hashing. I actually don't know whether Go's hashmap does the same.

From looking at the code[1], it seems like it does. From[2] it looks like it is only for platforms that have AES.

1. https://github.com/golang/go/blob/7bfe32f39c59056c49f5776b10...

2. https://github.com/golang/go/issues/9365

Hey! The great burntsushi himself answering me.

Thanks for taking the time to clarify specifics. I appreciate your attention.

I guess in Go strings are just a slice of bytes?

Isn't it possible to do the same with Rust for this benchmark?

Strings are just slices of bytes in Rust too. Both languages use UTF-8 as their internal representation. The difference is that Rust requires its strings to be valid UTF-8. Go does not. Either choice is reasonable. The downside of requiring valid UTF-8 is that in order to read bytes from a file, you have to do a validation check on them to ensure they are UTF-8 before returning a string back. This is an additional cost.

But this is not the only additional cost in the Rust program. I outlined a few others.

> Isn't it possible to do the same with Rust for this benchmark?

This benchmark has two programs for each language. The "simple" and "optimized" variant. The "simple" version is supposed to be written in an idiomatic style for that language. Taking extra steps to make tweaks and optimize the code is, I think, against the spirit of the challenge. Obviously, this is a very fuzzy concept, and everyone can make up their own mind on the extent to which this framing is useful. (I think it is, personally, especially when you also allow for a second submission that tries to make the program fast.)

To clarify my question, can Rust read files and split their strings without checking UTF8 correctness? That could speedup the algorithm.

In Go I have used this [1] in the past when I had to validate UTF8 encoding but on hot paths where I'm sure UTF8 is valid (coming from sanitized database data for example) I skipped that part.

[1] https://golang.org/pkg/unicode/utf8/#Valid

yes, you can operate on raw byte strings (you write them like b'this is ASCI' ), and then it would look like go or c code.

> To clarify my question, can Rust read files and split their strings without checking UTF8 correctness? That could speedup the algorithm.

That's what every single Rust program in this benchmark does, except for the simple variant.

Reply on news.ycombinator.com