benesch

> It affects performance too -- SQL wants storage locality by table, but applications often want locality by user.

This problem gets substantially worse in geographically distributed databases, like Spanner or CockroachDB, where joining tables whose data is stored in different localities can incur a significant latency penalty (in the hundreds of milliseconds).

They both offer an interesting solution to the problem called interleaved tables. CockroachDB docs on the subject are here [0], and Spanner's docs are here [1].

Interleaved tables don't fix the fact that SQL will hand you back a flat table when you have hierarchical data, but they do fix the spatial locality problem. Here's an example, from the CockroachDB docs, of how users and their order data can be interleaved, exactly the way it sounds like you've wanted:

    /customers/1
    /customers/1/orders/1000
    /customers/1/orders/1002
    /customers/2
    /customers/2/orders/1001
    /customers/2/orders/1003

[0]: https://www.cockroachlabs.com/docs/stable/interleave-in-pare...

[1]: https://cloud.google.com/spanner/docs/schema-and-data-model#...

Not unless you like pain. CockroachDB is distributed and uses Raft consensus under the hood, so you can't take a snapshot of just one node; you need to take a snapshot of all the nodes in your cluster, and you need to take those snapshots at the same logical instant. If the snapshots are from slightly different moments, when you boot the cluster up from the backup, the nodes will panic because you'll have corrupted the Raft state or otherwise caused the multiple replicas of the data to become inconsistent with one another.

There is one safe way to make this work: turn off your cluster, back it up with ZFS snapshots, then turn it back on. (Some of Cockroach's production tests do exactly this, in fact, because restoring from ZFS snapshots is so unbelievably fast.) But if you have the flexibility to power cycle your production database off and on in order to back it up... you probably don't need a distributed database, because fault tolerance likely isn't among your requirements.

(I'm a former CockroachDB engineer.)

JavaScript is an interesting case, in that it was the only way to write code that would run in Netscape Navigator at the time. In a sense it had Netscape behind it, and then it had Microsoft behind it once they reverse-engineered an implementation for IE3.

I'm not very familiar with the history of Python, though it seems it was funded in its early years by various research institutes that employed Guido van Rossum, rather than a giant corporation.

Yep, definitely. In this specific case of a missing unsafe marker you don’t need symbol versioning at all; you need the stdlib to somehow present two signatures for the same function that the compiler can select between according to the Rust version. It’s only for more complicated cases of API breakage (your function took an i32 but you need to extend it to an i64) that symbol versioning becomes necessary.

And great! All I wanted to point out was that, should the need arise, it is technically possible to tie stdlib function signatures to a particular edition.

Yes, absolutely! (Although note that I'm not the OP.)

With all due respect, I do think you're conflating the way things are with the way things have to be.

glibc has a similar commitment to never breaking user code, and yet symbols have been removed from glibc! The trick is to use "symbol versioning"; code compiled against old versions of glibc is transparently rewritten to use the old versions of symbols, while code compiled against new versions is rewritten to use the new versions of the symbols. [0][1]

You can imagine something similar for the Rust standard library, whereby both the unsafe and safe versions are provided, and packages are transparently rewritten to use the correct version based on their specified edition. You don't need two copies of the stdlib; just of the symbols that have diverged. In this case, you wouldn't even need two versions of the symbol in the resulting binary, since the addition of `unsafe` doesn't change the codegen of the function, but merely restricts what code is accepted by the compiler.

It's true that this would require some serious shenanigans in the stdlib, but it's a tractable problem. It's also, in my mind, quite acceptable for the standard library to play games like this, since the compiler, the language, and the stdlib are always going to be tightly coupled, and the complexity can be shielded from users.

All that said, it sounds like the list of deprecated stdlib features is nowhere near long enough to justify adding this kind of complexity.

[0]: https://gcc.gnu.org/wiki/SymbolVersioning

[1]: Note that symbol versioning is kind of a quagmire, but that's because it's seriously underdocumented and practically no developers are aware that it exists. The idea itself is sound.

Well hello, Dan! Nikhil here. It’s always nice to get a reminder that the internet can be a small place.

(Dan and I were at RC concurrently for the second half of his batch.)

I’ll echo the sentiment: RC is a really cool and unique place. I was somewhat atypical, I think, in that I came in with a very defined and niche project that I wanted to work on (hacking on gccgo), but the RC format is flexible enough to be amenable to that. It’s nice to be surrounded by a community of folks united by a love (or at least appreciation?) of programming, even if you’re not actively working on a project together.

I’m a bit sad that I missed out on some of the more social/collaborative side of RC by focusing so heavily on one project, but I’m told this is a common sentiment: it’s hard to strike a balance between being people-driven and being project-driven.

I'm not sure what you're arguing here. Yes, I described a very specific type of optimistic concurrency control that can be layered on top of Calvin to provide support for dependent transactions. (The "optimistic" is in the name of OLLP, after all.) But the underlying Calvin protocol is not optimistic, as it acquires locks. The result is a hybrid scheme that is neither fully optimistic nor fully pessimistic.

You can always execute dependent transactions in a system that does not support them by running two non-dependent transactions. This is covered in section 3.2.1 of the Calvin paper [0] as the Optimistic Lock Location Prediction (OLLP) scheme.

Basically, when you have a dependent transaction (i.e., a transaction where the write set depends on the results of earlier reads in the transaction), you split it into two separate transactions. The first transaction, the reconnaissance transaction, reads all the data necessary to determine the transaction's write set. Then the second transaction can declare the full read/write set based on the results of the recon transaction. While executing the second transaction, you verify that the data you're reading is the same as the data you read in the recon transaction; if it's not, you need to start the whole process over.

This is certainly unfortunate, because you have to perform every read twice. But perhaps it's performant enough. Have you tried using FaunaDB/Calvin for your workload? It sounds like FaunaDB has support for dependent transactions using OLLP baked in, but I'm curious to know if there's a significant performant hit to using it.

[0]: http://cs.yale.edu/homes/thomson/publications/calvin-sigmod1...

I've had entirely the opposite experience. Have you ever tried to install a package in Sublime? You have to use a separate tool, Package Control, whose installation instructions are arcane:

  import urllib.request,os,hashlib; h = '6f4c264a24d933ce70df5dedcf1dcaee' + 'ebe013ee18cced0ef93d5f746d80ef60'; pf = 'Package Control.sublime-package'; ipp = sublime.installed_packages_path(); urllib.request.install_opener( urllib.request.build_opener( urllib.request.ProxyHandler()) ); by = urllib.request.urlopen( 'http://packagecontrol.io/' + pf.replace(' ', '%20')).read(); dh = hashlib.sha256(by).hexdigest(); print('Error validating download (got %s instead of %s), please try manual install' % (dh, h)) if dh != h else open(os.path.join( ipp, pf), 'wb' ).write(by)

And once it's installed interaction is limited to the textual omnimenu-thing, since Sublime's extension API is extremely limited. Package Control is awesome for what it is, but it's severely constrained by not being a first-class feature of the UI.

In VSCode, by contrast, there's a proper UI for extension installation that's part of the core product. Clicking on the extension brings up a full-viewport window with the README and quick links to the reviews and GitHub. I've had a lot of luck finding quality extensions by installing the official Microsoft extension, if one exists, or otherwise using number of downloads/reviews/GitHub activity as a rough proxy for quality.

JetBrains IDEs have a very similar extension installation experience to VSCode (i.e., good), but I've anecdotally found the available JetBrains extensions to be lower quality than their analogous VSCode extensions. For example, I _still_ can't find a good comment-wrapping plugin for JetBrains—everything I've tried has some weird deficiency, like not being able to wrap comments at a different column than the code width, or sometimes butchering surrounding code, etc.

I'll admit that the one frustration I have with VSCode is that there is no way, at least not that I've found, to export your entire configuration, both settings and plugins, and use that to bootstrap a new VSCode installation. I recently set up a new computer and spent the first few weeks rediscovering all the extensions I'd installed on my old computer.

I'm curious to know if there's an editor/IDE that you think has a far superior plugin experience than VSCode.

CockroachDB engineer here.

> However key management in clusters is an absolute nightmare. They need to look at Docker Swarm for how to build a usable "join" mechanism.

I'll contend that key management is one of the great unsolved problems in software deployment.

I don't want to diminish the value of removing friction from the user experience, but fundamentally there is not that much of a difference between providing a token directly as a command line argument vs. providing a certificate that lives on disk. I'm curious if your frustration is mostly from the overhead of creating node certificates or if there's a deeper problem I'm not seeing.

I should also mention: this is an area we're actively looking to improve in! If you have time to write up a slightly more detailed proposal as a GitHub issue, we'd be happy to entertain it.

Release notes are available at: https://tip.golang.org/doc/go1.11

Headline features (as judged by me) are:

* A new WebAssembly backend, js/wasm, for the gc compiler.

* Preliminary support for modules, i.e., the first step towards a package manager built into the Go command.

* Improved support for debuggers, especially Delve.

* Removal of many direct system calls from the macOS runtime. Unlike on Linux, the macOS system call interface is not considered stable. Go 1.11 binaries are now less likely to break when you upgrade your macOS version because system calls are made through the proper channel (libc).

Ah, that is an unfortunate issue, and the error message is unreadable to boot.

> The issues pale in comparison to the horribleness of Glide, but it's interesting just how these tools end up being so damn flaky.

That's the crux of it, isn't it? The dozens of Go package managers that have come and gone over the years have provided us with substantial evidence that building a stable package manager requires several years of development. I think that's why I'm frustrated that the Go team hit the reset button again. Dep has accumulated plenty of bug fixes over the years to handle more and more of these edge cases, but vgo had to start from scratch.

On the bright side, vgo essentially can't fail.

> My personal theory is that what made Russ Cox cave in was his discussions with Sam Boyer. Cox thought Boyer was going down the wrong path, and thought he had a better solution.

That's certainly my understanding of the situation. Matt Farina has a great commented history of dep and vgo [0] if you haven't already seen it. The comments are particularly enlightening.

Still, it's not clear to me what made the Go team get into the package management game at all. As you say, for years they were happy to leave that as a community problem. But something spurred them to declare that Dep was an "official experiment."

> Dep is okay when it works, but inherits pretty much all the warts of Glide, which Boyer also worked on.

Funny, I've had exactly the opposite experience. Glide caused us plenty of trouble at CockroachDB, but Dep has worked flawlessly, if slowly. I've also found Sam to be exceptionally friendly and responsive to feedback [1] [2].

[0]: https://codeengineered.com/blog/2018/golang-godep-to-vgo/#co...

[1]: https://github.com/golang/dep/issues/1927

[2]: https://github.com/golang/dep/issues/460

I can't speak to this with absolute certainty, but I have some speculation to offer. The story of GOPATH is tightly intertwined with the story of package management.

Go is a Google project, and Google has a very unique approach to package management: commit everything to the monorepo. The GOPATH is, in essence, a monorepo. If you want to change the API of a library, well, you can just change all its callers across your GOPATH, too. And so for a long time the Go team was unconvinced that package management was a problem.

For example, the Go FAQ [0] still has this to say:

> How should I manage package versions using "go get"?

> "Go get" does not have any explicit concept of package versions. Versioning is a source of significant complexity, especially in large code bases, and we are unaware of any approach that works well at scale in a large enough variety of situations to be appropriate to force on all Go users...

> Packages intended for public use should try to maintain backwards compatibility as they evolve. The Go 1 compatibility guidelines are a good reference here: don't remove exported names, encourage tagged composite literals, and so on. If different functionality is required, add a new name instead of changing an old one. If a complete break is required, create a new package with a new import path.

It is true that if you write perfectly backwards compatible code, then you don't have a versioning problem, but if you think that's a viable solution you're ignoring certain realities of software engineering.

It wasn't until early last year that Russ Cox [1] publicly declared that versioning was a problem and set out to introduce a package manager into the Go toolchain. As it turns out, GOPATH is entirely incompatible with the approach to package versioning that the Go team settled on. You simply can't have two versions of the same package in your GOPATH, unless you're willing to rename one and rewrite all the import paths. Given that public opinion had turned again GOPATH [2], it was finally time to do away with it.

So it took about a year and a half from the time the Go team admitted GOPATH was a problem to shipping a release that made it unnecessary. That's really not too bad. The frustrating part of this saga were the first seven years during which the Go team refused to admit there was a problem at all.

[0]: https://golang.org/doc/faq#get_version [1]: https://research.swtch.com/go2017 [2]: https://github.com/golang/go/issues/17271

Before you get too excited about possible performance wins, note that gccgo turns out to be much slower than gc (the standard Go compiler) on a lot of real workloads. Here's one such benchmark, taken recently, though it's admittedly a microbenchmark [0]. Dave Cheney found similar results, though quite a while ago [1].

gccgo didn't support escape analysis for quite a long time, which meant that the performance cost of the increased heap allocations absolutely dwarfed whatever performance gain you got from GCC's smarter optimizations. I think it's recently gained support for escape analysis—it's actually quite difficult to find information about this, and whether you need a compiler flag to enable it, etc.—but I don't think that's going to tip the performance scales in gccgo's favor. EDIT: See below: 8.1 ships on-by-default escape analysis!

The primary motivations for gccgo, at least to my understanding, were a) having a second reference implementation that could find bugs in gc and the language specification, and b) support for esoteric platforms that will likely never make it into gc. The announcement of gccgo has more information about its motivation, though it makes some claims about performance that haven't stood the test of time [2].

[0]: https://groups.google.com/forum/#!msg/golang-nuts/mjTmIkWKZ6...

[1]: https://dave.cheney.net/2013/11/19/benchmarking-go-1-2rc5-vs...

[2]: https://blog.golang.org/gccgo-in-gcc-471

The constant thrash caused by the lack of a standard package management tool—even a de facto standard. Just look at the various package managers that have come and gone over the years:

1. https://github.com/tools/godep

2. https://github.com/kardianos/govendor

3. https://github.com/robfig/glock

4. https://github.com/rogpeppe/govers

5. https://github.com/Masterminds/glide

6. https://github.com/golang/dep

I'm sure I'm forgetting a few. And just when it seemed the Go team might be ready to standardize around Dep (6), they threw this wrench in the works.

I'm of two minds about vgo. I think it has some interesting ideas, but Dep works today, is widely adopted, and has nearly all the features you'd expect from a modern package manager.

My biggest frustration with the auto-update is how long it takes. I wouldn’t mind the auto-update in the slightest if it took 5s instead of 30s. It’s ironic (and a testament to your work on bottling) that the longest phase of installing a formula, even one with many deps, is waiting for `brew update` to complete.

I’m curious if you have plans to make `brew update` faster. Is it really just `git pull` that’s that slow?

Apologies for griping. Homebrew has made my life much, much better—thanks for all your hard work!

Simply drop the assembly as a .s file in a Go package and run `go build`. See Go's hash/crc32 package for a real example [0].

There's a lower level `go tool` command, of course, but I don't know it off the top of my head.

[0]: https://github.com/golang/go/tree/master/src/hash/crc32

I would love for this to be true, but I'm afraid I don't have any evidence that it is. What are the better options you'd recommend?

More specifically, Google's Bazel [0] is the most serious attempt I know of to replace CMake/autotools, but it only just began to support Windows [1] and requires the JDK. Say what you will about CMake, but it's a quick-and-easy MSI to install on Windows and has supported Windows for probably longer than I've been alive. Google's other open source build system, GYP [1a], supports Windows because it builds Chromium, but GYP might as well be undocumented. Unless you like spelunking into the Chromium source, GYP is a non-starter. (Props to the Node folks for actually doing this for Node and Node extensions.)

I haven't used Gradle [2], but I've heard more than a few times it's slow (warning: might be FUD)... and it also requires a JRE.

Ninja [3] is blazing fast, but it's specifically designed not to be used directly, but rather through a build system generator like CMake.

I'll be the first person to ditch CMake if a feasible alternative presents itself, but if you want to build a cross-platform, moderately-complicated C++ project today, especially one with dependencies on other libraries that use arbitrary build systems, I don't know of any such alternative.

[0]: https://bazel.build

[0a]: https://gyp.gsrc.io

[1]: https://docs.bazel.build/versions/master/windows.html

[2]: https://gradle.org

[3]: https://ninja-build.org

This is mentioned at the end of the article, but perhaps deserves to be called out more loudly:

> While CockroachDB was built to survive failures and prevent data loss, we want to make sure every team, regardless of size, has the ability to survive any type of disaster. Backup and restore were built for large clusters that absolutely need to minimize downtime, but for smaller clusters, a simpler tool will work just fine. For this, we’ve built cockroach dump [0], which is available in CockroachDB Core.

As you say, it's a hard line to draw, but our expectation is that any customer who has enough data that `cockroach dump` is infeasible is the kind of customer who is already quite familiar with enterprise licenses.

[0]: https://www.cockroachlabs.com/docs/stable/sql-dump.html

You might find FedEx's origin story interesting.

> “I asked Fred where the funds had come from, and he responded, ‘The meeting with the General Dynamics board was a bust and I knew we needed money for Monday, so I took a plane to Las Vegas and won $27,000.’ I said, ‘You mean you took our last $5,000— how could you do that?’ He shrugged his shoulders and said, ‘What difference does it make? Without the funds for the fuel companies, we couldn’t have flown anyway.’ Fred’s luck held again. It was not much, but it came at a critical time and kept us in business for another week.” [0]

[0]: http://www.huffingtonpost.com/2012/10/15/fred-smith-blackjac...

Out of curiosity, have you ever tried running barefoot? I spent four years of cross country practice in high school desperately fighting my tendency to overstride. At the start, my stride rate tended towards the high 70s (a healthy stride rate is high 80s, low 90s), and I barely got to the low 80s by the time I graduated.

The first time I went running barefoot, my stride rate was over 90. Instantly. If someone had told me, I would have been spared four years of stride rate exercises.

You might be right to claim that someone who's run barefoot their whole life will have no trouble running properly in standard runners, but if you don't already have perfect form cushioned shoes are not doing you any favors.

[Cockroach Labs engineer here.]

Yes, if very low-latency (i.e., P99 latency sub-5ms) reads and writes are critical to your application, CockroachDB should not be your first choice. That said, one of the primary motivations for CockroachDB is that most existing systems don't handle eventual consistency well. In our experience, most developers will eventually write code that assumes a consistent database, either accidentally or intentionally, because it works most of the time. Dealing with eventual consistency is hard.

Rather than "if you can work with eventual consistency, you should look elsewhere," the sentiment we're trying to cultivate is "if and only if your performance requirements can't work with strong consistency, then you should look elsewhere."

Yes, it's not just you. https://news.ycombinator.com/item?id=13811815

This appears to be an (illegitimate?) republishing of a course that Dan Armendariz has taught at the Harvard Extension School since 2008. (The course number has since changed from CS E-7 to Digital Media E-10.) You can find the fall 2015 materials available online for free, better-organized and without advertising, at http://digitalphotography.exposed.

If you find these materials helpful, or want feedback on your work, consider enrolling in the course through the Extension School [0]! Part of your tuition inevitably supports the Harvard bureaucracy, but Dan and his teaching fellows's stipends are directly tied to the number of students who enroll in the course, and they've kindly made the entire course available for free. You can also apply that course credit to an Extension School degree, which is worth more than whatever certificate of completion Alison is providing.

[0]: https://www.extension.harvard.edu/academics/courses/exposing...

Edit: I suppose I should note that the instructor for the current Extension School offering appears to have changed in the last year, so your mileage with the current version of the course will vary.

It depends how you define bloat. If you define bloat strictly as number of dependencies, then sure, the Node ecosystem encourages more "bloat" than any other mainstream ecosystem.

But if you define bloat as "inclusion of features I'm not using," the Node ecosystem is lean and mean. If I only need 15 of the 250 functions that Lodash provides, I can include each of those 15 functions as an individual package. Yes, it increases disk space, because each dependency ships a package.json and a README and subdependencies aren't shared, but disk space is cheap. Decreasing the surface area of my library dependencies is worth it for my development happiness.

I think the criticism you want is "ultra-modular." You can make an argument that Node's proliferation of tiny, one-function packages is a maintenance burden and causes discoverability problems. Personally, I disagree: any effort to move towards a world where you never have to write a line of code someone else already has is well worth it. And from an end-user perspective, it shouldn't matter at all. `npm install` on average takes no longer than `gem install` for me, despite NPM pulling in orders of magnitude more packages.

(Python might be bloated; have you ever looked at the complete stdlib? There are some horrifying pieces: urllib2, email, turtle (?!).)

P.S. Check out libsass [0], a re-implementation of Sass in C with [probably] bindings to your favorite programming language.

[0]: https://github.com/sass/libsass

Shameless plug: I had the exact same question at my last gig, and wrote up a quick open-source tool to build wheels and dump them to an S3 bucket. [0]

Usage is as simple as

    mkwheelhouse mybucket.mycorp.co scipy numpy

which will automatically build and upload those wheels for your current architecture and dump them into mybucket.mycorp.co. It builds a pip-compatible index, too, so you can just tell pip to search your wheelhouse first before falling back to PyPI:

    pip install --find-links mybucket.mycorp.co scipy numpy

If you need to build for several OSes, you can run mkwheelhouse against the same bucket from each OS.

The downside of this approach is you can't host private packages, because you need to enable public website hosting. (Although, VPC endpoints might have changed this!) But the simplicity of this approach plus the massive speedup of not needing to constantly recompile scipy was totally worth it.

[0]: https://github.com/whoopinc/mkwheelhouse

I'm with you, but don't get your hopes up. Verizon never wired the city for FiOS because it was "too costly" [0].

Though something I always wanted to try when I lived downtown was netBlazr [1], which beams internet from antennas on the tops of buildings. If you've got your own place or can convince your landlord to install an antenna, they get fantastic reviews.

[0]: http://www.bostonglobe.com/business/2013/10/08/wahlberg-fios...

[1]: http://www.netblazr.com/residential/

Heh, I'm a computer science major to my core. Spoken like a CS major at a liberal arts school, I suppose.

I suspect we agree. If you could universally define "better life," you could quantify it. But you can't. We're also bad--really bad--at analyzing ourselves. Right now, I'd love to be spending 60 hrs/week programming and less time writing papers. But I bet I'll feel differently in a decade.

> And no. College is not optimizing its graduates for a better life. College, since its inception, has assumed that you came in with one.

I'm not sure what you mean. Higher education has become increasingly accessible to the economically disadvantaged, not less.

Also, here's Harvard College's mission statement:

> Harvard College adheres to the purposes for which the Charter of 1650 was granted: “The advancement of all good literature, arts, and sciences; the advancement and education of youth in all manner of good literature, arts, and sciences; and all other necessary provisions that may conduce to the education of the … youth of this country….” In brief: Harvard strives to create knowledge, to open the minds of students to that knowledge, and to enable students to take best advantage of their educational opportunities. To these ends, the College encourages students to respect ideas and their free expression, and to rejoice in discovery and in critical thought; to pursue excellence in a spirit of productive cooperation; and to assume responsibility for the consequences of personal actions. Harvard seeks to identify and to remove restraints on students’ full participation, so that individuals may explore their capabilities and interests and may develop their full intellectual and human potential. Education at Harvard should liberate students to explore, to create, to challenge, and to lead. The support the College provides to students is a foundation upon which self-reliance and habits of lifelong learning are built: Harvard expects that the scholarship and collegiality it fosters in its students will lead them in their later lives to advance knowledge, to promote understanding, and to serve society.

That's pretty damn well summarized as "we empower our students to lead better lives." We can argue over whether college comes close to achieving that goal, but I'm comforted by the fact that the stated mission at least aligns with my personal reasons for attending school.