HNHacker News
TopNewBestAskShowJobs

YorickPeterse

2,784 karma · joined March 29, 2011

Author of the Inko programming language, formerly a staff developer at GitLab.

Website: https://yorickpeterse.com

submissionscomments
YorickPeterse··on Show HN: Inko – A safe and concurrent object-oriented programming language

    > Looks pretty good. I see someone has been reading "The Error Model" by Joe
    > Duffy, eh?
Yes! It was a _huge_ inspiration. In fact, I wasn't sure what error handling mechanism to use until I read the article.

    > Something like "public_send" or "define_method". And maybe show runtime
    > reflection capabilities, if Inko has them.
Good call, I'll note this down. Technically this is possible at the moment, but it's not exactly user friendly.

Regarding your list of suggestions, I agree that having those would be nice, and I'll see how I can best add those.

The state of the standard library can best be described as "very young". For example, only recently did I manage to add enough building blocks to build a testing standard library, and one of the next goals is to make sure everything is tested before adding more.

Benchmarks aren't available yet because I think it's too early. FFI is planned, but not yet supported.

YorickPeterse··on Show HN: Inko – A safe and concurrent object-oriented programming language
Author here. Both the language and website are still very much a work in progress, so I'm happy to answer any questions here.
YorickPeterse··on How not to structure database-backed web apps: performance bugs in the wild
http://sequel.jeremyevans.net/
YorickPeterse··on Moving from Azure to Google Cloud Platform
A year or two ago Y Combinator startups would be given around $1M in Azure credits, should they move to Azure. GitLab.com at the time moved over to Azure, with the goal being to reduce costs before we consumed those credits. I think we consumed all of it in a matter of months. Once you're stuck with a hosting provider (Azure or not), it's difficult to move away from it.
YorickPeterse··on Gitlab 500 error
> Or make it automatic when the database isn't available?

I agree this is probably a good idea, so I created https://gitlab.com/gitlab-com/infrastructure/issues/4382 to keep track of things.

> were there any signs of impending doom before the event happened that could have warned us, and do we have alerts on that?

I don't think you can detect corrupt WAL until it is actually corrupt. In those case the host won't start up, for which we already have various errors that cover that case (e.g. there's an error for when too few transactions are happening).

There also wasn't really any impending doom before we started things. Two issues that triggered this were:

1. A lack of `set -e` 2. Apparently when we stop a command using `gitlab-ctl`, it will time out if the process doesn't stop fast enough. I'm not sure what exit status we report in that case, but in this instance it resulted in the script continuing to run.

YorickPeterse··on Gitlab 500 error
Yes that's a typo, it should have been `set -e`
YorickPeterse··on Gitlab 500 error
> Speaking of postmortems, reading through the Google Doc of their outage timeline, my first postmortem question would be why is the website dependent on the database being up?

You can't do anything with GitLab without the database being available.

> Shouldn't there at least be a fallback that says "We're offline right now sorry" when the database is unavailable? Or in the name of extreme transparency, have it frame or show a link to the google doc of the outage?

We have a deploy page, but it requires us to explicitly enable it, which can be easy to forget. I do think adding a link to our Twitter status to the error pages could be useful, or maybe a widget of some kind.

> My second question would be how often do we test failures of a read slave, of the master, and of pgbouncer?

In the past we tested certain scenarios whenever necessary. For example, when introducing new parts of the database load balancer we'd test what happens if we terminate X or Y. However, we do not yet do periodic chaos monkey style testing, which has been on our todo list for quite a while.

> Do we at least have a practiced procedure for manual failover if not an automatic failover?

Failovers are actively worked on (https://gitlab.com/gitlab-com/database/issues/91, https://gitlab.com/gitlab-com/database/issues/86, https://gitlab.com/gitlab-com/database/issues/40). Right now our failover procedure is still quite painful, and it's easy to make mistakes. We held off using this approach for our maintenance because of these reasons, and also because a failover usually results in 2-3 hours of slow loading times. Ironically we ended up with just that.

> My third question would be what sort of alerts do we have, if any, on replication lag?

Plenty: https://gitlab.com/gitlab-com/runbooks/blob/810824765ec45534...

In this particular instance they go to a Slack channel called "#database", and this channel has been on fire with alerts since the problem started.

YorickPeterse··on I’m Nat Friedman, Future CEO of GitHub. AMA
That's something we don't have any data on.
YorickPeterse··on I’m Nat Friedman, Future CEO of GitHub. AMA
Out of the 250 000-ish projects that migrated to GitLab.com, only about 3000 are mirrors (= using GitLab's mirroring feature).

Source: I work for GitLab.

YorickPeterse··on OrmHate (2012)
I have always found ActiveRecord to be extremely limiting, and Sequel to be much better. Some random limitations I have had to deal with over the years (in no particular order):

* No support for "OR" statements, without dropping down to raw SQL (I think this was added in either Rails 4, or Rails 5)

* Terrible support (if any) for more advanced PostgreSQL features, such as CTEs, bigserial (Rails 5 finally supports this if I'm not mistaken), no proper support for composite primary keys, no support for PostgreSQL CHECK constraints, no support for index expressions (e.g. an index on `lower(foo)`), etc. For GitLab we had to add many hacks to work around this.

* A rather messy model API, where models are used as both repositories and objects representing single rows. Sequel makes the same mistake with its ORM, but at least you can use the query toolkit separately. ROM supposedly does this better, but I have not tried it.

* Too many methods are injected into your classes, and you'll probably never use most of these (this is a more philosophical issue, and certainly not a deal breaker).

There are plenty more, but these are the ones that come to mind. My biggest issue in recent times was caused by ActiveRecord/Arel silently throwing away CTEs used in UPDATEs. Basically in GitLab we did something like this (I don't fully remember what the query was):

    WITH foo AS ( ... )
    UPDATE some_table 
    SET a = b
    WHERE x = foo.y
In other words: we'd grab some data using a (recursive) CTE, then update a bunch of rows based on that data. Unfortunately, ActiveRecord/Arel would throw away the CTE in certain cases. This would result in us effectively running the following:

    UPDATE some_table
    SET a = b
You can imagine the fun of having to deal with all rows being updated unexpectedly. More details on this particular bug can be found here: https://gitlab.com/gitlab-org/gitlab-ce/issues/37916

Long story short, I'm fairly convinced that using Sequel would have made GitLab's database code much more solid/less buggy than it is today, simply because it supports so much more (and better) out of the box.

YorickPeterse··on Lisp, Jazz, Aikido
This might differ between countries and schools. I practised it for several years, and we had many classes that involved techniques useful for avoiding conflict (some as simple as just running away), keeping people at distance, and so on.
YorickPeterse··on Lisp, Jazz, Aikido
Self defence techniques such as Krav Maga teach you the exact same, while also teaching you how to actually defend yourself in a variety of scenarios.
YorickPeterse··on Lisp, Jazz, Aikido
Something not being used in MMA doesn't necessarily mean it's completely useless, it just means it doesn't fit MMA. The sport isn't static, instead it changes as new ideas and techniques come and go.

If my memory serves me right, Jiu-Jitsu wasn't used in the professional (western) fighting scene for quite a while. Today it's pretty much a must-have for MMA.

The same goes the other way around. Being an expert in Jiu-Jitsu means squat when your opponent is holding a gun, and standing 10m away from you. In other words: different flavours of martial arts/self-defence have different applications, though I would certainly consider some to be very ineffective in most situations.

YorickPeterse··on Lisp, Jazz, Aikido
The key is to also understand that it will never help you in a fight of any kind.
YorickPeterse··on Managing database schema changes without downtime
This is not how we do it. Constraints, columns, etc are removed using a post-deployment migration. This means you can't end up with NULL values and the likes.
YorickPeterse··on Managing database schema changes without downtime
At GitLab we do the following:

We have two migration directories: db/migrate and db/post_migrate. If a migration adds something (e.g. a table or a column) or migrates data that existing code can deal with (e.g. populating a new table) then it goes in db/migrate. If a migration removes or updates something that first requires a code deploy, it goes in db/post_migrate. We combine this with a variety of helpers in various places to allow for zero downtime upgrades (if you're using PostgreSQL). For example, to remove a column we take these steps:

1. Deploy a new version of the code that ignores the column we will drop (this is a matter of adding `ignore_column :column_name` in the right class).

2. Run a post-deployment migration that removes said column.

3. In the next release we can remove the `ignore_column` line (we require that users upgrade at most one minor version for online upgrades).

The migrations in db/post_migrate are executed by default but you can opt-out by setting an environment variable. In case of GitLab.com this results in the following deployment procedure:

1. Deploy code with this variable set (so we don't run the post-deployment migrations)

2. Re-run `rake db:migrate` on a particular host without setting this environment variable.

For big data migrations we use Sidekiq to run them in the background. This removes the need for a deployment procedure taking hours, though it comes with some additional complexity.

More information about this can be found at the following places:

1. https://docs.gitlab.com/ee/update/README.html#upgrading-with...

2. https://docs.gitlab.com/ee/development/what_requires_downtim...

3. https://docs.gitlab.com/ee/development/background_migrations...

YorickPeterse··on GitLab 10.4 released
Do you have any examples on what kind of pages are loading slow? Are these issue detail pages, MR diff pages, etc? It's possible you're running into cases we're not experiencing (enough) on GitLab.com, or maybe we did but we haven't really looked into them yet.
YorickPeterse··on GitLab 10.4 released

    > Yeah, and they've promised to work on performance for ages now with almost
    > no improvement.
That's simply not true, there have been a _ton_ of improvements that we made over the past 2 years. A very simplified example:

A specific GitLab.com issue in December 2015 vs January 2018 (I can't seem to find what the exact URL is):

2015: http://stats.gitlab.com/1902794/2015/12

2018: http://stats.gitlab.com/1902794/2018/01

Apart from that you can take a look at any of the past release posts or merge requests tagged with "performance" [1][2] and you'll see that plenty of improvements have been made over time.

    > Is ruby really such a mess that it's impossible to run an app with
    > reasonable memory consumption?
Ruby is not really to blame for this, instead it's mostly Rails and all the third-party libraries that we add on top that consume so much memory.

[1]: CE improvements: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests?scope...

[2]: EE improvements (some of these may be merges from CE): https://gitlab.com/gitlab-org/gitlab-ee/merge_requests?scope...

YorickPeterse··on PostgreSQL HA cluster failure: a post-mortem
PostgreSQL's replication itself is perfectly fine, albeit a bit bare-bones (e.g. it's just replication plus the ability to trigger a failover). For something like cluster management and automated failovers you'll need e.g. https://repmgr.org/.
YorickPeterse··on Coinbase Ordered to Turn Over Identities of 14,355 Crypto Traders to the IRS
The Cayman Islands?
YorickPeterse··on Mozilla terminates its deal with Yahoo and makes Google the default in Firefox
Probably the answer is quite simple: Google has more money to offer.
YorickPeterse··on Debian and GNOME announce plans to migrate communities to GitLab

    > "testing in production" indeed. One of my repos has been stuck in limbo for over 2 months now 
    > after I attempted to migrate it to a group namespace from my own profile.
Have you created an issue for this on https://gitlab.com/gitlab-com/support-forum/issues?
YorickPeterse··on Debian and GNOME announce plans to migrate communities to GitLab
You are most likely getting downvoted because of statements such as:

    > These guys are not honest people, just copycats under the guise of "open".
YorickPeterse··on Scaling the GitLab database
We've had connection pooling on the client side since forever, just like any other Rails project. This article specifically talks about server side pooling.
YorickPeterse··on Scaling the GitLab database

    > Buy into NoSQL hype, push 5x resources on your NoSQL solution
NoSQL is so last year, NewSQL is the future!
YorickPeterse··on Scaling the GitLab database
You can have perfectly valid input escaping without needing prepared statements. Many ORMs handle this without the need for PREPARE/EXECUTE.
YorickPeterse··on Scaling the GitLab database
There was an issue with the dashboard causing it to try and load data from the wrong place, this has been fixed. See https://gitlab.com/gitlab-com/infrastructure/issues/3115 for more info.
YorickPeterse··on Scaling the GitLab database
Yes, you can typically push PostgreSQL very far while still using a fairly simple setup (e.g. no sharding). Unfortunately too many times people have this mindset that a slow application is the result of a slow/bad database (as in "it's an RDBMS and RDBMS' don't scale"), and not the result of it being misused (e.g. badly written queries, lack of proper indexes, that sort of thing). At GitLab it took quite a while to get everybody to see this.
YorickPeterse··on Scaling the GitLab database
It would require a complete rewrite for at least the data layer, as everything right now assumes an RDBMS. Further it would complicate installation as now everybody would have to use a fancy "NewSQL" database, some of which you may not even be able to run yourself.
YorickPeterse··on Scaling the GitLab database
Most of our problems thus far have been the result of misuse of our resources one way or another. GitHub on the other hand probably dealt with these issues earlier on (or are able to hide them somehow, perhaps by throwing a lot of physical hardware at the problem), thus they can now focus on dealing with problems caused by reaching the limits of their (old) systems.
← PreviousPage 10 of 14Next →