3,383 karma · joined August 30, 2011
Twitter: https://twitter.com/brandur
GitHub: https://github.com/brandur
Website: https://brandur.org
Email: brandur@brandur.org
There is still some performance cost for checking a response against a schema though, so a nice compromise is just to check a small % of outgoing responses — the vast majority of requests stay as fast as possible, but given a reasonable traffic load, it's still enough to eventually reveal any places the schema doesn't match reality. (This is an approach we use at Stripe for checking our OpenAPI specification.)
There's a good chance that we'll see this somewhat proven in that Apple will probably get that box discontinued within the next couple weeks.
That is unless they were trying to make it look sudden, but I don't think we have good enough reason to be that cynical about it.
I have nothing against dependency files, but yes, it's probably even more obvious.
If I see an `import "github.com/x/y"` at the top of a Go file, I know that I can go find the source for that at `cd $GOPATH/src/github.com/x/y` (unless the project has a `vendor` directory in which case you'll look in there instead).
> What dependency management system doesn't allow you to do this if you insist?
I'm sure you could do it on anything, but Go makes it very easy.
As noted upthread, it's pretty simple to do this in other languages like Ruby when you're using Bundler and know the system well (e.g., `pushd $(bundle show excon)`), but in many languages it's much more difficult. You'd need to go chase down the source, download it, then redirect your dependency to point to it. Sometimes you'll grabbed a newer version that's no longer compatible so you'll have to make sure to work off of the right tag.
In bad cases (e.g., C, Java) you'd have to learn how to build the project which might not be trivial because tooling isn't standardized or convenient to use.
Also, having helped a number of people now through their early days of the language it's also the one biggest thing by far that reliably confuses every single person. And I mean everyone — from first time programmers all the way up to people who've been in the industry for decades and are learning Go as their tenth language. Being forced to put files in certain places is incredibly non-intuitive because there's nothing else out there that requires it. I must have sent the Go documentation on workspaces [1] to two dozen different people at this point.
But on the other hand, once you've grasped the system and are using it, `GOPATH` is surprisingly not bad. It's always obvious where your dependencies are located and which versions are going to be used to build your project. Even better, it lets you very easily drop into those dependencies and add minor changes or debugging lines if you need to. This can be incredibly useful if you're trying to understand how one of them works or think that that you might have found a bug and trying to verify or patch it. A very powerful feature once you know about it.
The new Go modules seem good, and will be a huge improvement in lowering the barrier to entry for Go, but I'll miss the old `GOPATH` style of work at this point.
I'm using Google Domains right now, but have been using CloudFlare to host my DNS for ages for this reason alone. I'll think about transferring my domains to them when the time comes to take one service out of the equation.
[1] https://support.cloudflare.com/hc/en-us/articles/200169056-C...
https://www.postgresql.org/docs/current/static/functions-jso...
From the docs for the `||` operator:
> Concatenate two jsonb values into a new jsonb value
And for `jsonb_set`:
> Returns target with the section designated by path replaced by new_value, or with new_value added if create_missing is true ( default is true) and the item designated by path does not exist. As with the path orientated operators, negative integers that appear in path count from the end of JSON arrays.
You'd use this function in conjunction with an `UPDATE` statement to get the result that you want.
A few notes from taking a look at the code:
* I've very much started doing the same thing in my projects of just using modern JavaScript constructs like fetch and arrow functions without a transpiler. IE is never going to support any of them, but I've accepted that. Caniuse.com ranks most of them at 85%+ supported.
* It's neat to see that the IEX API supports batch requests (as opposed to having to pull one stock at a time) in its API which makes it possible to implement the project efficiently and in a way that doesn't have to store state.
* Nice work building it to support large users as well. A lot of installations probably won't have more than 100 stocks, but it'll break up requests into multiple batches of <= 100 if they do.
Some programming languages can be, and are, better than other programming languages. They've all been evolving along a single historical timeline, and many mistakes were made in the past that we acknowledge and wouldn't make again. To take a well-known example: eminently abusable Turing complete templates in C++ versus the more modern generics as seen in C#, Java, Rust, etc.
We should all have a shared goal of building of tooling that maximizes our leverage by providing powerful primitives, and minimizes the number of mistakes we make by providing dependable safety rails. Some more recent programming languages are really doing an exceptional job of this.
This critique of Go quite correct. It's a pretty good language with a lot of pretty good ideas, but it's also not without a healthy dose of very sizable flaws as well (which the author has enumerated nicely), and they're made more concerning because most of the Go team refuses to acknowledge many of them.
Without even going near the third rails of package management or generics, other niceties like real enums, ways to build typed non-slice non-map data structures, safety measures to protect against Goroutine leaks, or explicit interfaces would all be big improvements. I personally find that once it comes to large projects, Go feels obviously less brittle than Ruby/Python/JS, but much closer to their end of the spectrum compared to languages that offer stronger guarantees (e.g. Rust, Haskell, but also even languages like C#). I don't think that's what its creators were going for.
Will do! For what it's worth, I also had `chan` in dependencies list, but didn't list it above because it's one of the minority where the docs and examples were very good.
More generally, I feel quite bad for complaining about things instead of going in to improve them (through bug reports or patches), but there has to be a balance between making forward progress and stopping to help shore up the tooling. That said, I haven't been doing enough of the latter lately, so I'll make more of an effort.
> For one example; we're working really hard right now on getting async/await shipped, and dealing with ergonomics problems around it. The recent changes are to make it simpler, not make it harder. For example, you should need to think a lot less about future composition, since you'll be able to borrow across futures, as just one example. > > There might be a disconnect between what we're working on and what we're perceived to be working on. Messaging is hard!
Sufficed to say that I can't wait to see these improvements come out. It may end up being the case that I simply got started on this project a little too early — six months from now many of my problems may have evaporated already.
I'll quickly point out that Rust is my favorite programming language, even if reading this piece in isolation would give you the opposite impression. There are about a million things to like about it: The syntax, type safety, sober choices around language design, built-in documentation facilities, conventions, `rustfmt`, linting with `clippy`, toolchain management with `rustup`, community (in the sense of project management and organization), development momentum, ecosystem, and a whole host of other things are all downright incredible. Some of the people working around the core are undoubtedly some of the smartest throughout the entirety of professional software.
That said, I wouldn't take back anything I wrote here because even when I read it back today with a much more optimistic place (I bypassed many "brick walls" that I alluded to since I wrote it), I still think it's true. Months later, even after writing a lot of Rust, I often still don't feel productive.
No language is perfect, and it's probably healthy to have the occasional counterpoint to its fairly consistent positive press. I've spoken to people who don't know much about the language and refer despairingly to the "Rust Evangelism Strikeforce", which I think is largely internalized rationalization that what they've read is too good to be true, so there must some nefarious element at work. Rust really does deserve its good press, and hopefully the occasional dissenting opinion and subsequent discussion will help convince them of such.
IMO, Rust has a few existential threats — the one that I think most about is that despite being well passed 1.0 at this point, there's a distinct lack of pragmatism around getting core features like concurrency nailed down and shipped. Futures and Tokio are widespread at this point, but the APIs are still changing, and the error messages that they produce are still quite awful. Even once they're fully feature-complete and stable, they're still not going to be very pleasant to use — you have to think really hard about how you're doing future composition. Contrast this to a concurrency system like Go's, where you can more or less pretend that you're writing synchronous code, and let the runtime take care of the heavy lifting. I love how much careful thought and consideration is being put into the development of a cohesive concurrency model, but unless it can get to the point where it's stable and far more user-friendly, people (like me) who are trying to use the language to build things are going to continue having a hard time.
In summary, Rust is awesome, the core team is awesome, and the development community is awesome. I'm confident that the usability problems it has today will be worked out.
I wrote this piece hastily, and it really wasn't intended for this broad of an audience — I won't redact the existing wording I still think it's roughly right, but I do regret a lot of it.
Rust's docs are amazing in certain contexts — the book is great, the built-in support for documentation on types/functions/etc. is amazing, and compiling code examples are a very laudable idea.
What I'm speaking to here is more once you get into the broader ecosystem and start using a lot of non-core libraries. Oftentimes I found that the front page docs explaining the basic premise were concise and well-written, but that things got quite a bit harder when you started diving into individual classes and functions (put another way, as you started deviating from the happy path). This doesn't apply everywhere, but often the comments are very minimal and the documentation relies heavily on "types as documentation" in that there's a big list of all the traits and the functions on those traits that are implemented. In many, many cases there's little in the way of detail or examples.
I've written quite a bit of Rust now and have used many of the headliner projects. So far there have been very few crates where I didn't have to resort to eventually checking out the source tree and figuring out how to do some particular thing by examining its source code and test suite. I won't call out any single project in particular, but I found this to be the case in every one of `actix`, `clap`, `diesel`, `error-chain`, `horrorshow`, `hyper` and `juniper`, just to pick a few from the top of my `Cargo.toml` (it also happened with many other libraries). It's great that you can do this and open source is awesome, but ideally I could get by on just documentation, which is what you can do in many other languages.
Unfortunately, read with little context, it sounds like the tone of my piece was intended to crucify, but it's not. It can be simultaneously true that Rust's docs can still use lots of improvement and that the Rust team is doing an amazing job of improving them (there's just a lot of work and a long way to go). Both these facts are true with Rust.
https://news.ycombinator.com/item?id=16169803
The short version is that I don't think it's fair to compare Go's stability to that of a "barebones" language with an infinitesimal stdlib (e.g., C), because that's where most of the change is likely to happen. Go's already proven to be far more stable than most "batteries included" languages: think Python 2 vs. 3, Ruby's fairly considerable churn over the years, PHP 4 to 5, etc.
Obviously my designation on Go here is subjective, but it seems to me that it compares favorable to any other platform that comes to mind.
> But for a fair comparison you'd need to include a lot of C libraries to match Go's standard library. How much have those libraries changed over the years?
Yes, exactly. Something like C isn't a fair comparison because there's so little included in the core language. Think about everything that you get with Go's standard library: concurrency, HTTP (even HTTP2), TLS, JSON, XML, Zip encoding, Regexp, etc., and then consider C's APIs along with all the APIs of all the libraries you'd need in C to accomplish those tasks — all of that together would be a more fair comparison.
That might seem like apples and oranges, but for most real-world programs, people generally need a considerable subset of that functionality. C's stability is nothing to scoff at, but it's nowhere near as impressive of an achievement.
I think this is a pretty uncharitable interpretation of what I said. I didn't say "the most", but rather "one of the most". Kubernetes, Docker, and Chromium are all huge projects with similar or higher commit rates, but think about the other 99.99% of other open-source projects. In comparison to them, anything of Go's scale is huge.
Try not to focus on the individual wording though. If you like, just read it as "a very active project".
> Also, instead of updating the `users` table with the LSN of the commit - which creates extra write load - why not store it in the session cookie, then you can route based on that.
The article's a little long, but at one point I do mention that we're putting this information in Postgres in the demo for convenience, but that in a real implementation it might be worthwhile moving it elsewhere as an optimization.
This addresses the first point as well: you're right in that this information can be procured from Postgres, but the point of putting in the observer is to demonstrate a system that could be implemented in a way that's agnostic of storage. It's very plausible that you might want to put `min_lsn` and replication statuses in something very fast like Redis, and even have each `api` worker caching its own version of the latter so that a replica can be selected without even making a network call.
I'm storing `min_lsn` directly to the database, so anything connecting to it and loading a user record should be able to take advantage of the information.
I have a note in there about this as well, but although I'm showing how this information can be persisted to Postgres so that I'm only using one data store, it could really live in any kind of shared persistence layer.
For example, replication statuses and a user's `min_lsn` might be put in Redis instead, and be treated as ephemeral data such that we don't worry about losing them because they can be repopulated on every restart. It's also probably a good idea to have individual API workers caching replication statuses directly in memory so they can select a read candidate without even having to go to a database.
Yeah, I'm relying on transactional semantics to guarantee no contention during the lock, but the part that I'd intended as important here is really just the idea of locking the key — not the particular implementation. I think alternatively using a `FOR UPDATE` is totally plausible and likely even to be more performant than what I've written here.
Try to imagine this as more of a "ideas guide" rather than a dictation on precise implementation.
But even so, theaters have been amazingly resilient to the digital age. I'd chalk this up to the fact that they provide a shared social movie-going experience that's not quite something you can replicate at home (i.e. it's something to go do with your friends), and in some cases, you want to see certain hits with a certain timeliness so that you can talk about them at school/work/with friends.
Hardcovers are just not like this at all. Even if you'd compare them favorably to an ebook, you'd be hard pressed to find one person on Earth who would say that they provide a superior experience over a paperback (except that they look more handsome on a bookshelf). They are, and always have been, quite simply a way for the publishing industry to squeeze a few extra dollars out of consumers who were willing to pony up the money so they could read a book early. This is exactly the sort of wrinkle that you'd expect to be ironed out in the Information Age — people's tolerance for this kind of profiteering decreases as access to media broadens.
The article's author is right that diminishing the edge of hardcovers is certain to cut into the margins of bookstores like his own, but he's wrong that it's a bad thing (directly). It might cost him money, but it's better for literally every single book buyer.
One thing that I probably should have been clearer on: I used Sidekiq as a queue example here, but this pattern generalizes to anything. RabbitMQ is just as plausible.
> Background jobs are just one part of what a robust message queue gives you. In my opinion, the desire for background jobs is a design smell that indicates a flaw in your architecture which you can fix by adding a message queue system.
Possibly ... one thing to consider is that a fair number of us are writing various types of web services, and in web services there are so many obvious tasks that should be moved out of band to a background job. It's not even so much about distributing workload (although that's a nice aspect) as it is about moving expensive operations out-of-band so that a user's request finishes faster.
Here's a couple examples:
* Process an uploaded image into a variety of thumbnail sizes.
* Fire a webhook.
* Send a welcome email.
* Duplicate added/updated information to an internal microservice.
* Start a file download (like synchronizing a remote RSS feed).
In all these cases there's no reason for the request that initiated the action to wait on the job being finished. Moving the work to the background is purely a win from the user's perspective.
You need to be a little careful about configuration, but believe it or not, as long as Redis is configured with an append only file [1] it will provide you with a durability guarantee just like Postgres would. If it crashes, any jobs that were in there are recovered the next time it starts up.
From the docs:
> From now on, every time Redis receives a command that changes the dataset (e.g. SET) it will append it to the AOF. When you restart Redis it will re-play the AOF to rebuild the state.
---
Good question. I should have probably put a comment to annotate that.
The transaction is there purely to wrap the `SELECT` statement, then later the `DELETE`. If it fails midway through, the process will restart, and you will get doubled up jobs inserted into your queue. This isn't very desirable, but it's better than the alternative, which is `DELETE` the jobs too early and then lose jobs when your process dies.
> If you want at-least once message delivery, can't you configure that in your queue? (For example, RabbitMQ supports both modes; only ACKing after the task completes, or ACKing as soon as the worker dequeues the message).
Your queue will also be configured to do this (i.e. in the case of Sidekiq, it won't fully free a job until it's been confirmed to have succeeded or failed on at worker), but you basically need to have at least once delivery between any two systems. So the enqueuer will hand jobs off to a queue with this guarantee, and the queue will then hand them off to its own workers.
> I'm not familiar with Sidekiq, so maybe that's not an option there. But in that case, it's still not clear why you'd requeue all the tasks if one of them fails to be enqueued (or the delte fails); you could just decline to delete the row for the individual job that failed.
Yeah, keep in mind that this a pretty naive code sample for purposes of simplicity. You can and should be more clever about it like you suggest if you're putting it in production. However in practice, enqueuing a job is likely to have a very high success rate, so probably this simple worker will do the trick in most cases.
Using `LISTEN`/`NOTIFY` is certainly workable. It's worth considering though that once you've got non-trivial traffic, you're going to have new jobs showing up all the time, so the loop is likely going to be cycling endlessly no matter what mechanic you use to feed it.
At the very least though you'd want a sleep statement with exponential backoff so that it's not churning away on an empty database at the low end.
Yeah, I'd forgotten to mention that in the article, but yep, something should be guaranteeing that only one enqueuer is running at a time.
> Knowing that this easy transition exists makes me even more confident in just using Que and not adding another service dependency (like Redis) until it’s really needed.
Yeah, so many stacks have a Redis for something these days anyway that you can probably start with it in a lot of cases, but if you're running really lean that seems like a good strategy.
Totally. This also leads to other operational tricks that you hope you never need, but are great the day you do. For example, a bad deploy queues a bunch of jobs with invalid arguments which will never succeed. You can open a transaction and go in and fix them in bulk using an `UPDATE` with jsonb select and manipulation operators. You can then even issue a `SELECT` to make sure that things look right before running `COMMIT`.
Again, something that you hope no one ever does in production, but a life saver in an emergency.
I've taken fire before for suggesting that any job should go into a database, but when you're using this sort of pattern with an ACID-compliant store like Postgres it is so convenient. Jobs stay invisible until they're committed with other data and ready to be worked. Transactions that rollback discard jobs along with everything else. You avoid so many edge cases and gain so much in terms of correctness and reliability.
Worker contention while locking can cause a variety of bad operational problems for a job queue that's put directly in a database (for the likes of delayed_job, Que, and queue_classic). The idea of staging the jobs first is meant as a compromise: all the benefits of transactional isolation but with significantly less operational trouble, and at the cost of only a slightly delayed jobs as an enqueuer moves them out of the database and into a job queue.
I'd be curious to hear what people think.
I guess the answer is the same as how the country is able to win gold medal at the Olympics [1] – money (and lots of it) sunk into state-sponsored training programs — but I'm a little surprised that it's effective. Even in the US, a school that would consistently produce A-grade hackers would be pretty hard to build.
---
[1] https://en.wikipedia.org/wiki/North_Korea_at_the_Olympics
This is essentially the same problem that we see reoccurring everywhere in modern society. We put in a system that seemed like a good idea, but after some time being exercised in reality, turns out to have some serious flaws.
Fixing those flaws is so time consuming and expensive that it becomes a bureaucratic impossibility. The result is that the profits pass onto the few, and everyone else suffers for it. (e.g. politician buying in Washington, software patents, prop 13 in California, etc.)