Hacker News Reader

Aside from project's reason, there's one very good reason to re-implement a bunch of stuff in Rust: testing whether it delivers. It's being pushed as a safer, better systems language. So, let's take many different C/C++ apps, rewrite them in Rust, and see what the results are across many metrics. By the end of it, we should know Rust's strengths and weaknesses in real-world coding.

The README answers this:

  To learn Rust.

Edit: It also mentions not being tied to UNIX and appears to claim it will run on Windows. That's certainly something.

Sorry if I wasn't clear, but I am looking for a more general answer! I would like to know in which case it is useful (to other people) and discuss it's value comparatively to writing interfaces to other languages.

I had no intention of making the end result useful, but I run into interesting problems.

First, I wanted to make it as pure Rust as possible. I tried to avoid UNIX specific code, and since there is no library with Windows support for asynchronous IO in Rust, I was pushed into spawning thread and blocking waiting for client data. I quickly noticed that the benchmark was way below Redis (around 60% of ops/sec with 50 clients). But then someone point out to me[1] that I was running tests in a machine with two cores, and this actually may be better for machines with multiple cores[2]. I have yet to try it out and benchmark the results.

So far Rust API was disappointing for network operations. For example, `TcpStream.read()`[3] and `TcpListener.incoming()`[4] do not have a timeout. Maybe because its development is driven for Servo and not for servers.

I have thought about doubling down on multithreading and instead of a global database lock as rsedis is using now, having one per key (or some other arbitrary partition), and having concurrent operations, which is hard to do safely in C. But I have not gotten there yet.

[1] https://github.com/jonhoo/rucache/issues/2

[2] https://github.com/jonhoo/volley/

[3] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpStream...

[4] http://doc.rust-lang.org/1.0.0-beta/std/net/struct.TcpListen...

Your comment is interesting to me because although i've only looked at Rust from a distance, its seemingly lack of features related to server side code ( threading , non blocking io and networking api) have made me wait before i actually start coding in that language.

Seems like those parts still need more work.

It depends on what you're doing. See https://github.com/jonhoo/volley, which uses blocking IO and 1:1 threads, and we're still damn fast.

There are also libraries like mio that can give you no blocking IO. Rust's standard library is deliberately minimal, exploring the ecosystem a bit is helpful, though admittedly not as easy as it could be. Working on it!

Right. People should realize that there is no latency difference between blocking and non-blocking IO -- in fact there is a small difference in blocking IO's favor. The difference is in throughput when there is a very large number of concurrent requests, namely more than about 10K and certainly more than 2K. If you're serving under 2K concurrent requests, blocking IO is probably a better choice than nonblocking. Of course, the number of concurrent requests you need to serve depends on the time you spend serving each request; Little's Law and all that.

    > Maybe because its development is driven for Servo 
    > and not for servers.

This is not true at all. If Rust's development were determined by Servo, we would have kept green threads and implemented struct inheritance by now.

The reason timeouts were dropped is in the IO/OS reform RFC: https://github.com/rust-lang/rfcs/blob/8fa971a670f9b9bc30f31...

    >  set_timeout has been removed for now (as well as other
    > timeout-related functions). It is likely that this may
    >  come back soon as a binding to setsockopt to the 
    > SO_RCVTIMEO and SO_SNDTIMEO options. This RFC does not
    >  currently proposed adding them just yet, however.

And on UDP:

    > All timeout support is removed. This may come back in
    > the form of setsockopt (as with TCP streams) or with 
    > a more general implementation of select.

I'm on shaky wifi and my phone, so I can't find a citation for this, but I also believe it was removed due to 1) Rust not having any stable representation of time and 2) needing to shim certain behaviors on some platforms, which we decided wouldn't happen in the first round of stable interfaces.

That said, the lack here certainly hurts, and we did manage to stabilize Duration, paving the way for timeouts to return.

EDIT: Oh! I forgot that https://github.com/rust-lang/rfcs/pull/1047 got merged recently. https://github.com/rust-lang/rust/issues/25818 implemented it. http://doc.rust-lang.org/nightly/std/net/struct.TcpStream.ht... shows it implemented on nightly, so you can actually even do timeouts today, just not on the stable channel.

Thanks for the clarification. I wanted to subscribe to rust's internals debates and proposals, but I was not sure how to find them. Should I be looking at https://github.com/rust-lang/rfcs or is there anywhere else?

There's a few stages:

1. We keep open issues on that repo to track ideas.

2. At some point, someone may decide to formally propose an idea. They may or may not post a "pre-RFC" to internals.rust-lang.org to get early feedback.

3. An actual RFC will get filed at that repo as a PR.

4. The relevant sub team will check it out and comment, and at some point, consensus will be reached either way.

5. The RFC will go to 'last call' for a week, making sure all voices have been heard.

6. Assuming nothing in last call blocks moving forward, the RFC will be accepted.

7. The RFC will be implemented.

So, in short, yes, subscribing to that repo will notify you of the relevant discussions.

You can also subscribe here: https://internals.rust-lang.org/

The main places to look are https://internals.rust-lang.org/, https://github.com/rust-lang/rfcs, https://github.com/rust-lang/rust (for actual implementation), and the Rust IRC channels such as #rust-internals on irc.mozilla.org.

IRC is used for quick casual conversation. The internals forum is used for discussion, "pre-RFC", and the like. The rfcs repo is use for actually discussing formal RFCs, as well as wishlist language and library bugs. The rust repo is for the actual implementation of rustc and the standard library itself.

>Could someone (or OP) elaborate on the value that re-implementing a whole software to a new language provide comparatively to just building an interface "bridging" both worlds?

Making it safer and even catching bugs in the original implementation (both things Rust will help with)?

Making it integrate seamlesly with the new language's ecosystem? E.g. Lucene is Java, and someone could use that, but there are tons of ports of it in C, C++, Python etc, providing convenience to integrate it with projects in these languages.

>And if I am not mistaken you would need a team at least as large as the parent project to implement new features, fix bugs and keep pace with it.

Not necessarily. A project with 10 part time contributors could be matched with a project with 2-3 full time competent hackers for example, or even surpassed.

> Lucene is Java ... there are tons of ports of it in C, C++, Python, etc.

There used to be several ports, though most are dead and/or are several major versions behind. A new C++ or Rust port would be great, though unrealistic given the huge project side.

>though unrealistic given the huge project side.

It could be done with some automated translation, and cleared out from there.

This project specifically is probably not useful for others just because it is written in another language (unless it were to succeed in fixing problems or improving security or performance.) Redis is a server application, not a library, so there's a clearly defined and interoperable protocol to bridge the Redis server to other languages already. Writing a Redis client for the other language would be more useful.

More generally, as coldtea mentions, making integration into the rest of the language's ecosystem is the primary benefit of rewriting in another language.

The value of such a port to others depends on how easy it is to integrate between the two languages, either via libraries or other methods. The harder it is to integrate the two (and the absence of automated translation tools) increases the value of the rewrite to others.

Your Core-NLP example is actually an interesting one, because that library has already been ported to other languages... It is available for the C#/F# ecosystem (http://sergey-tihon.github.io/Stanford.NLP.NET/).

In this case, it's a learning exercise. Sometimes, it's because there are other benefits, such as making it easier to distribute or easier to tie into specific parts of the algorithm than may be possible by calling a library that provides a high level interface.

From a learning perspective, reimplementation has key advantage versus other kinds of projects: the design is completely done, so you can focus exclusively on the mechanics of implementation.