HNHacker News
TopNewBestAskShowJobs

YorickPeterse

2,784 karma · joined March 29, 2011

Author of the Inko programming language, formerly a staff developer at GitLab.

Website: https://yorickpeterse.com

submissionscomments
YorickPeterse··on How is team-member-1 doing?
This is mostly spread across different issues in our issue tracker (https://gitlab.com/gitlab-com/infrastructure). I suspect we'll write up a blog post once all the moving parts are in place, have been tested/used for a while, etc.
YorickPeterse··on How is team-member-1 doing?
The shirt is internal only, we're not selling it to the public.
YorickPeterse··on GitLab acquires Gitter, will open-source the code

    > However, the service (gitlab.com) is constantly having issues, most of
    > them not reported on their status page or on their twitter status account. 
    > For the last week it's been practically unusable, to the point where our
    > whole dev team combined has wasted almost a hundred hours just re-trying
    > builds and deployment jobs.
Do you happen to have some examples of these problems? I recall there are some known problems with CI runners getting stuck, but I'm wondering if you were running into other problems as well.
YorickPeterse··on Postmortem of database outage of January 31
I haven't received any offers so far. I don't intend on leaving GitLab any time soon either.
YorickPeterse··on Postmortem of database outage of January 31

    > 1. notifications go through regular email. Email should be only one
    > channel used to dispatch notifications of infrastructure events. Tools
    > like VictorOps or PagerDuty should be employed as notification
    > brokers/coordinators and notifications should go to email, team chat, and
    > phone/SMS if severity warrants, and have an attached escalation policy so
    > that it doesn't all hinge on one guy's phone not being dead.
This is discussed in https://gitlab.com/gitlab-com/infrastructure/issues/1095. We're basically looking into moving to Prometheus monitoring combined with Pagerduty notifications.

    > 2. there was a single database, whose performance problems had impacted
    > production multiple times before (the post lists 4 incidents). One such
    > performance problem was contributing to breakage at this very moment. I
    > understand that was the thing that was trying to be fixed here, but what
    > process allowed this to cause 4 outages over the preceding year without
    > moving to the top of the list of things to address?
High availability has been a thing we wanted to do for a while, but for whatever reason we just never got to it (until recently). Not sure exactly why.

    > Wouldn't it be wise to tweak the PgSQL configuration and/or upgrade the
    > server before trying to integrate the hot standby to serve some read-only
    > queries?
The server itself is already quite powerful, and the settings should be fairly decent (e.g. we used pgtune, and spent quite a bit of time tweaking things). The servers currently have 32 cores, 440-something GB of RAM, and the disk containing the DB data uses Azure premium storage with around 700 GB of storage (we currently use 340).

    > And since a hot standby can only service reads (and afaik this is not a
    > well-supported option in PgSQL), wouldn't most of the performance issues,
    > which appear write-related, remain? The process seriously needs to be
    > reviewed here.
Based on our monitoring data we have vastly more reads than writes. This means load balancing gets very interesting. Hot standby is also supported just fine out of the box, you just need something third party for the actual load balancing.

    > And am I reading this right, the one and only production DB server was
    > restarted to change a configuration value in order to try to make
    > pg_basebackup work?
We suspect so. Chef handles restarting processes and we think it's currently still set to do a hard restart always, instead of doing a reload whenever possible.

    > What impact did that have on the people trying to use the site a) while the
    > database was restarting
A few minutes of downtime as the DB is unavailable.

    > and b) while the kernel settings were tweaked to accommodate the too-high
    > max_connections value?
No. We now reduced the max_connections to a lower value (1000) so we still have enough but don't to tweak any kernel settings.

    > Is it normal for GitLab to cause intermittent, few-minute downtimes like
    > that? Or did that occur while the site was already down?
We've had a few too many cases like this in the past. We're aiming to resolve those, but unfortunately this is rather tricky and time consuming.

    > 3. Spam reports can cause mass hard deletion of user data? 
Yes.

    > Has this happened to other users?
Not that I know of.

    > What's the remedy for wrongly-targeted persons?
A better abuse system, e.g. one that makes it easier to see _who_ was reported. We're also thinking of adding a quorom kind of feature: to remove users more than 3 people need to approve it, something like that.

    > And is the GitLab employee's data gone now too?
No. The removal procedure was throwing errors, causing it to roll back its changes. This kept happening, which prevented the user from being removed. So ironically an error saved the day here.

    > How could something so insufficient have been released to the public
Code is written by developers, and developers are humans. Humans in turn make mistakes. Most project removal related code also existed before we started enforcing stricter performance guidelines.

    > and how can you disclose this apparently-unresolved vulnerability? By so
    > doing, you're challenging the public to come and try to empty your database
There's no point in hiding it. Spending a few minutes digging through the code and you'll find it, and probably plenty other similar problems. If somebody tries to abuse it we'll deal with it on a case by case basis.

    > because LVM snapshots now occur hourly, and that it only takes 16 hours to
    > transfer LVM snapshots between environments :)
LVM snapshots are stored on the host itself. As such if e.g. db1 loses data we can restore the snapshot in a few minutes. They only have to be transferred if we want to recover other hosts. Furthermore, in the Azure ARM environment the file transfer would be much faster compared to the classic environment.

    > 4. the PgSQL master deleted its WALs within 4 hours of the replica
    > "beginning to lag" (<interrobang here>). That really needs to be fixed.
Yes, which is also something we're looking into.

    > Again, you probably need a serious upgrade to your PgSQL server because it
    > apparently doesn't have enough space to hold more than a couple of hours of
    > WALs (unless this was just a naive misconfiguration of the
    > [min|max]_wal_size parameter, like the max_connections parameter?)
Probably just a naive configuration value since we have plenty of storage available.

    > There were a few other things (including someone else downthread who pointed
    > out that your CEO re-revealed your DB's hostnames in this write-up, and that
    > they're resolvable via public DNS and have running sshds on port 22), but
    > these are the big standouts for me.
Revealing hostnames isn't really a big deal, neither is SSH running on port 22. In the worst case some bots will try to log in using "admin" usernames and the likes, which won't work. All hosts use public key authentication, and password authentication is disabled.

    > Not sure how fast your disks were, but 300GB gone in "a few seconds" sounds
    > like a stretch.
Nope, after about 2 seconds the data was gone. Context: I ran said command.

    > Some data may've been recoverable with some disk forensics.
When using psycial disks not used by anything else, maybe. However, we're talking about disks used in a cloud environment. Are they actually physical? Are they part of larger disks shared with other servers? Who knows. The chance of data recovery using special tools in a cloud environment is basically zero.

    > Especially if your Postgres server was running at the time of the deletion,
    > some data and file descriptors also likely could've been extracted from
    > system memory
That only works for files still held on to by PostgreSQL. PostgreSQL doesn't keep all files open at all times, so it wouldn't help.
YorickPeterse··on Postmortem of database outage of January 31
Aha, I probably overlooked read replicas (haven't used RDS in about 1.5 years).
YorickPeterse··on Postmortem of database outage of January 31
RDS, or any hosted database solution, is not some kind of silver bullet that solves all problems. While it's true it takes care of backups automatically, it does also restrict you in terms of what you can do.

For example, you can't load custom extensions into RDS. Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries, and replication between RDS and non RDS is also not supported. This means you can't balance load between multiple hosts, unless you're OK with running a multi-master setup (of which I'm not sure how well this would play out on RDS).

Most important of all, we ship PostgreSQL as part of our Omnibus package. As a result the best way of testing this over time is to use it ourselves, something we strive to do with everything we sihp. This means we need to actually run our own things. Using a hosted database would mean we wouldn't be using a part of what we ship, thus not being able to test it over time.

YorickPeterse··on Our long term plan to make GitLab as fast as possible with Vue and Webpack
There is no such thing as:

1. Replace Ruby implementation

2. ???

3. Everything is 3x faster

Whatever an implementation or product claims, it's just not possible. While we could benefit from not having a GIL (something Truffle has if I'm not mistaken), it's not a free trade-off. For example:

* JRuby typically requires quite a bit more memory compared to CRuby, the same probably applies to Truffle

* Dependencies might have to be replaced if they're written in C. I personally don't buy Truffle being able to support all extensions. It's super fucking hard. I spent a few years working on Rubinius and had the unfortunate experience of having to deal with C extensions a lot. Supporting these properly is really, really difficult.

* JRuby might not work on all platforms we support. For example, we support Raspberry Pi's and the likes and it's not clear if JRuby would work there

Most importantly of all: a large portion of our slowness is due to badly written SQL queries, code that hammers the database way too often, and our Git operations. None of this would be solved by switching to a different Ruby implementation. The only way to solve this is to:

1. Solve the code

2. Educate developers so they don't make the same mistakes

3. Add linting rules and the likes so our CI environment catches any mistakes (where possible)

YorickPeterse··on Postmortem of database outage of January 31
Still here, and doing just fine.
YorickPeterse··on GitLab’s Secret to Managing Employees in 160 Locations: Write Everything Down
There's a work in progress merge request for the post mortem, which can be found at https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/....

This blog post mentions a whole bunch of issues that are being worked on, this list can also be found at https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108.

> Furthermore, I won't be surprised if they have seen more concentrated spam attack, after the news of this data loss surfaced.

Not yet. Since Akismet has been enabled for snippets it seems the amount of spam is gone, or at least not noticeable.

YorickPeterse··on Gitlab is down
It's still being written: https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/..., in the mean time quite a bit of work is underway, see the issues mentioned in https://gitlab.com/gitlab-com/www-gitlab-com/issues/1108 for more info.
YorickPeterse··on Our long term plan to make GitLab as fast as possible with Vue and Webpack
We have no plans to move to another Ruby implementation, primarily because Ruby itself is not yet a bottleneck for us.
YorickPeterse··on Data Loss at GitLab
As mentioned below we only used Slony to upgrade from 9.2.something to 9.6.1. For regular replication we use PostgreSQL's streaming replication.
YorickPeterse··on Data Loss at GitLab
It's both fun and incredibly frustrating at times.
YorickPeterse··on GitLab Database Incident – Live Report
I'm not sure if it's the CEO's mistake, or any specific individual's mistake for that matter. In this particular case many different problems coalesced, producing one giant problem. If it wasn't me, somebody else would have eventually made the same mistake; perhaps with even greater impact.
YorickPeterse··on GitLab Database Incident – Live Report
I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.
YorickPeterse··on GitLab Database Incident – Live Report
I don't think we use PITR, certainly not to S3. I believe wal-e was discussed internally some time in the past, but we never really did anything with it.
YorickPeterse··on GitLab Database Incident – Live Report
issues, merge requests (titles, descriptions, etc), comments, events ("X pushed to Y"), labels, projects, milestones, users, permissions, project statistics, CI builds information, abuse reports, the list goes on.
YorickPeterse··on GitLab Database Incident – Live Report
A regular user that can use `sudo`.
YorickPeterse··on GitLab Database Incident – Live Report
Correct, I'm based in Europe/Amsterdam so this happened mostly during the evening.
YorickPeterse··on GitLab Database Incident – Live Report
It's not just the tech stack, but a combination of the technical choices made and with the human procedures behind them. We're actively pushing towards getting everybody to focus on scalability, but there's still a lot of debt to take care of.
YorickPeterse··on GitLab Database Incident – Live Report
We have snapshots, but they're not very recent (see the document for more info). The most recent snapshot is roughly 6 hours old (relative to the data loss). The data loss only affects database data, Git repositories and Wikis still exist (though they are fairly useless without a corresponding project).
YorickPeterse··on GitLab Database Incident – Live Report
Nope, that would be me.
YorickPeterse··on GitHub lost $66M in nine months of 2016
> I don't understand why they wont just throw more servers at the problem

There's only so much you can solve by throwing more hardware/money at the problem. We have reached a point where we are wasting too much of this, so adding more won't help much.

> (or put some devs on fixing their backend)

We have plenty of people working on the problem, and for quite a while now. We're also hiring more developers to help us out with this:

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests?scope...

https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/...

https://gitlab.com/gitlab-org/gitlab-ce/issues/25421/

There's also this old (and closed) issue which contains a lot of information: https://gitlab.com/gitlab-com/operations/issues/42

YorickPeterse··on Proposed server purchase for GitLab.com
Puma is something we want to eventually run, but right now it's not really a priority. Instead we're focusing on response timings and memory usage. See the following links for more information:

https://gitlab.com/gitlab-org/gitlab-ce/issues/3592

https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1899

YorickPeterse··on How We Knew It Was Time to Leave the Cloud
Not all users may use Amazon or S3. This means that if we were to build something requiring S3 we'd cut off those users.

Building some kind of generic storage layer with multiple backends (S3/S4/etc) would result in us only maintaining whatever we happen to use for GitLab.com. It could also complicate maintenance too much.

YorickPeterse··on How We Knew It Was Time to Leave the Cloud
> They probably could have saved themselves a lot of pain by talking to some Ceph experts still working inside RedHat for architectural and other design decisions.

We have been in contact with RedHat and various other Ceph experts ever since we started using it.

> I agree with other poster who asked why do they even need a gigantic distributed fs and how that seems like a design miss.

Users can self host GitLab. Using some complex custom block storage system would complicate this too much, especially since the vast majority of users won't need it.

YorickPeterse··on Dutch prison crisis: A shortage of prisoners
Dutch citizen here, this is bullshit. Neighbourhood watches using WhatsApp has nothing to do with a lack of police force, it's simply an "add-on" so neighbours can report suspicious behaviour to others.

> Within ten years these groups will probably grow into armed and organised right-wing groups.

You miss the part where we have very strict gun ownership rules, and very few have guns.

> Thanks for the down votes without knowing the facts.

You are being downvoted because what you provide aren't facts. You linked to a news article about neighbourhood watches, an article about a group of extremists/xenophobes, and an article about Dutch vs German law. None of this proves that The Netherlands is the a "dangerous" country where citizens take matters into their own hands, etc.

YorickPeterse··on GitLab 8.13 Released with Multiple Issue Boards and Merge Conflict Editor
Plenty of work is being done in terms of performance improvements, and almost every release features a large set of improvements. A list of all performance related merge requests that have been merged can be found at https://gitlab.com/gitlab-org/gitlab-ce/merge_requests?scope....

If you'd like to help out you can read up on the contributing guidelines at https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBU....

Alternatively, any issues with data on what is slow and how these patterns can be reproduced is also greatly appreciated!

YorickPeterse··on GitLab reinstates list of servers that have malware
This depends a bit on what parts of GitLab(.com) you're using. For example, viewing issues should generally be as fast as GitHub.com (http://stats.pingdom.com/81vpf8jyr1h9/1902794/2016/10 vs http://stats.pingdom.com/81vpf8jyr1h9/1902795/2016/10). Other parts may be a bit slower; pushing for example can still be a bit slow at times.
← PreviousPage 12 of 14Next →