Jonhoo

Thanks! Glad to hear that you found it interesting :)

It was, in fact, originally posted at https://news.ycombinator.com/item?id=11773332 by me (the author), but that post never got much attention :p

For self-referential datastructures (your first point above), using an Rc or Arc shouldn't have any overhead. I agree that it would be nice to be able to express this, but it's not really that big of a problem.

For partial borrows, there has been some work on it (https://github.com/rust-lang/rfcs/issues/1215), but I agree that this is something that's missing. That said, I very rarely run into this, and there's usually some fairly obvious restructuring I can do to make it work out.

I think it'd actually be good to show both to show how versatile Rust can be in this regard. I added it to the article with a link back here. Thanks!

In my defense, the next sentence is "Unless you explicitly mark your code as `unsafe`, your code simply cannot have data races." That said, I've updated the article text to now say data races in both places.

I barely, if ever, use/need a REPL. The shell has all the tools I need for short, one-off tasks.

I don't have any numbers to support this claim, but it's certainly true that we are seeing a lot of software being developed in Go, and network software in particular.

You can still do dependency injection in Rust..? There even exists a create for it: https://github.com/Nercury/di-rs. There's also some further discussion here: https://users.rust-lang.org/t/how-do-you-implement-dependenc...

Actually, no. I've found that I'm maybe even more productive in Rust than I am in Python, even for one-off tasks. The one exception is probably plotting, but I'm sure a Rust library for that isn't far off.

I agree with your point in that this is written in an excessively functional style. Something more akin to the imperative-style solution given by lifthrasiir below might have been simpler. However, I don't think that code shows that Rust is more expressive in any way than C/Go is (which is kind of the point).

The functional code I give in the article is quite radically different from what you would/could do in those languages, and I would argue that it does show that Rust provides some interesting and flexible mechanisms that add to the expressivity you have as a programmer. Sure, it's overused in the example, but the exaggeration at least has a chance of communicating the point. Maybe the argument is that I should have used a simpler (yet still not trivial) example instead, but I couldn't think of one at the time.

Re the "Not only is this very readable" part, I completely agree. I could change it to something less, hmm, bold, but I'm not sure it would make that much of a difference at this point. Suggestions are welcome.

I'm slightly hesitant to agree with both of these. The first seems wrong to me, because Bash qualifies, whereas Tcl doesn't. The second is better, though I think it excludes some interesting languages, like Scala, which you probably wouldn't write an operating system in, but that you could certainly build "systems" in (e.g., Spark).

To some extent, I think this comes down to familiarity with the functional approach to solving these kinds of problems. Certainly, before I started writing code in Rust, I also preferred the imperative approach you would take in C and Go. Over time though, I've found that I increasingly prefer this way of expressing chained computation.

That said, there's certainly a point to be made (and indeed other commenters have made it already) that this isn't code you would want to maintain. For production-level code, it'd be broken down more, so that individual pieces could be reviewed and tested in isolation. The code example here was to demonstrate the expressivity of the language more so than the One True Way of doing it.

Of course. This example was given to show expressiveness. For production-level code, you would break it up more so pieces can be tested individually (as pointed out in another comment).

Not entirely sure what you mean here? Even Bash can handle (some) OS signals, but I don't think anyone considers that a systems language?

Author of the post here. Curious that this got posted again. Was originally posted as https://news.ycombinator.com/item?id=11773332. Can the posts be merged by a mod somehow?

That is indeed how I meant it. That said, Go is certainly being used as a systems language, and especially in the domain of networked services, it is rapidly replacing C/C++.

If you compare that to many of the other solutions on Rosetta Code (https://www.rosettacode.org/wiki/Inverted_index), that's a pretty neat solution! And yes, as pointed out below, rustfmt would clean this up (mostly by increasing the indentation). I reduced indent to double spaces to make the code denser for the post.

Try again. Just did a re-deploy to fix a typo.

fwiw, already posted here: https://news.ycombinator.com/item?id=9681717

It is worth noting that the profiler is fully open-source: https://github.com/plasma-umass/coz. There is even a package for Arch Linux!

Hehe, thanks, fixed.

I never refer to it as a tree, only as, effectively, a tiered mesh. While it's true that this breaks down with multi-hop BGP forwarding, I think it conveys enough about how the routing works for a layperson to understand that is relatively close to the truth, no? I could potentially cut down the section on Internet routing significantly, and simply mention that it is a routing fabric that does relatively greedy hop-by-hop forwarding, though I'm not sure that will be more digestable.

You're right that BGP doesn't care about hostnames, or even individual IPs, but I'm not sure that distinction is actually relevant to someone who is being exposed to this just now.

No, I disagree. IP has no multiplexing features beyond protocol multiplexing. Port numbers that allow multiplexing a single IP among multiple applications on the same host only appear in UDP/TCP. What you are referring to is specifically called HTTP multiplexing, and has little to do with TCP.

Again, technically, you are right, however there is nothing in SMTP that precludes the sender from contacting the destination SMTP server directly (well, except that spam detection systems will freak out). The fact that SMTP supports hop-by-hop forwarding, and in many cases this is the only mode that is used, does not mean that this is necessitated by the protocol. And in the interest of making the content easier to understand, I decided describing multi-hop SMTP was simply unnecessary.

I appreciate the feedback --- it's always hard to piece together relatively complex posts like this. As I mentioned elsewhere, I specifically wanted this to be a single post such that readers could follow the communication flow end-to-end. I believe this will improve reader comprehension, though of course YMMV.

I see your point, but I think it's still important to give an end-to-end description, allowing people to tie it all together. The wall of text, if you'd read it, uses progressive layers in several places (like core Internet routing), specifically to try and avoid information overload for the reader.

If a reader were to actually use networks for something (like try to write a program that uses TCP), they would obviously need much more detailed knowledge. At that point, diagrams become necessary, as do much deeper explanations, but the goal of this post was to give an overview of what things are needed to provide end-to-end communication, and why.

I'd be happy to get rid of it if it is indeed out-dated as you say, though that was not my impression. Could you give me some references giving information about how core Internet is now being done?

No no, there are torrents for all of these videos (at least usually). However, torrents are slightly less convenient in that you can't play the file until the download completes. For me personally it's also an upside that HTTP is easier to proxy, and there's no need to seed. That said, torrents will probably be a better solution for many people. If you're interested, I wrote a tool a while back that scanned torrent sites for the newest episode of a particular show and automatically downloaded and started the torrent: https://github.com/jonhoo/smore. No idea if it still works, but shouldn't be too hard to get it up and running again.

Yes, the second method I used to use quite frequently, but it means you have to open your browser, click play to get the video to load (which will spawn a torrent of ad pop-ups and captcha prompts and whatnot). This way I can get the download straight from the command line.

Author here: for some sites, that approach works well, and this is what I originally did (and why I decided to write streamsh in bash). Have a look in the "tricks" directory and you'll see some of the much simpler solutions that are tried first. However, an increasing number of sites try to hide the URL, and through increasingly complicated means. The article is about vidbull.com, but there are others. They often also layer multiple obfuscators (vidbull uses both JS obfuscation and URL encryption for example). While it's true you can get pretty far just by scanning for mp4s on some mirrors, they are becoming fewer and farther between.

stream.sh <video URL (preferably the embed URL)> <filename to download to (without extension)>

FWIW, this has now moved to https://github.com/jonhoo/drwmutex so you can use it directly in your projects.

You'll acquire CPU 0's lock (since a map lookup with an invalid key yields the zero value). I agree this isn't optimal when you change the affinity of a process after it has started. You could imagine a scheme where, if this happens, you create a new lock, but that would significantly complicate the scheme as you would now potentially need to take a read lock on the map in case it changes under you. It's annoying that CPUID values aren't guaranteed to be without holes, but that's what we're stuck with.

Yeah, the sleep is a leftover from an earlier version of the code that didn't use sched_setaffinity. I've remove it now.