Comment by tabbott | Hacker News Reader

Comment by tabbott | Hacker News Reader

No team is capable of writing complex software without eventually introducing security bugs, and we (Zulip) are no exception. I'm disappointed that we introduced this bug, and would like to apologize to our users.

As a bit of commentary, my experience is that most SaaS vendors fix security bugs quietly. Others will do an brief internal investigation and only consider doing a public disclosure if they believe data privacy regulations require them to do so, because they discovered that the vulnerability had been in fact exploited. Usually public disclosure decisions are made by business executives with PR considerations in mind, so that shouldn't be surprising. See, for example, https://www.zdnet.com/article/slack-resets-passwords-for-1-o....

Zulip's core values include transparency, and our policy is to publicly disclose all security bugs that we discover with a formal CVE number, and this blog post is part of that practice. As noted in the blog post, we did an extensive audit and emailed every customer who our audit logs could not prove was not affected with enough detail to let them audit the situation. We did that despite our audit finding no evidence that this vulnerability was ever intentionally exploited, because we think it's the right thing to do.

Some technical detail I can add for those interested is that our postmortem plans following this incident include further investment in the internal audit log system used for the investigation (`RealmAuditLog`, if any readers feel like grepping our codebase). Improvements we made in mid-2020 were very helpful in investigating whether this bug had in fact been exploited, and we take this as a sign that we should increase investment in that system.

As a side note, I'm surprised that this specific security disclosure ended up on the Hacker News homepage, while our security release about an RCE issue affecting self-hosted Zulip servers last month did not (https://blog.zulip.com/2022/01/25/zulip-server-4-9-security-...).

The core bug there an insecure secret generation algorithm in upstream Erlang/RabbitMQ, which seems more broadly notable to me!

Despite our spending many hours composing detailed reports and lobbying security teams to get this problem fixed for everyone, not just Zulip, `apt install rabbitmq-server` continues to make a Debian/Ubuntu system vulnerable to RCE, and this fact has apparently been public for years.

We did manage to get `rabbitmq-server` fixed in Debian testing, and it is scheduled to be included in the next Debian stable release, I would really have thought an RCE issue like this would result in an immediate security advisory.

As a final remark, if you self-host Zulip, please subscribe to our announcements mailing list (https://groups.google.com/g/zulip-announce) and make sure your installation is up to date, so that you benefit from the hard work our security team does to keep Zulip users safe.

The blog post tries to paint the RabbitMQ cookie generation as a security issue, but the Erlang distribution cookie is NOT a security feature. Indeed, the Erlang documentation states:

> "Security" here does not mean cryptographically secure, but rather security against accidental misuse, such as preventing a node from connecting to a cluster with which it is not intended to communicate.

The purpose of the cookie is to guard against _accidental misuse_, such as a user connecting to the wrong node (prod vs testing?). The cookie is not meant to secure a system against unauthorised use. Any system relying on the cookie for security should already be considered broken. Thus I can see why there has not been such hurry to "fix" the issue.

That argument would hold more water if there were an additional security mechanism in place preventing `apt install rabbitmq-server` from exposing a system to a remote code execution. It seems to be that the simplest fix for this problem is to make the cookie secure, since doing so is likely effective with minimal effort. Another option would be to have clustering disabled by default.

If you'd like to summarize the upstream issue as "RabbitMQ as well as every distributors of it that we've investigated allows remote execution of arbitrary code on your server through having no effective access control mechanism for the distribution port, which is open by default after installing the software.", that summary is defensible.

That still sounds to me like a critical security issue. One should be very disappointed in a vendor that had not patched years after its public disclosure, especially given the context that we're apparently not first project using RabbitMQ to be affected by these security choices by Erlang and RabbitMQ. See, for example, https://nvd.nist.gov/vuln/detail/CVE-2018-1279, back in 2018. We definitely won't be the last, either.

I hope nobody is going to argue that it's reasonable to publish server software with a "documented" RCE vulnerability, especially if you don't make any effort to highlight that detail. https://www.rabbitmq.com/access-control.html, the main page about access control in RabbitMQ, does not even mention this security concern; you have to read their clustering guide, which should be irrelevant to anyone not intending to use RabbitMQ clustering.

> RabbitMQ as well as every distributors of it that we've investigated allows remote execution of arbitrary code on your server through having no effective access control mechanism for the distribution port, which is open by default after installing the software.

This is exactly how I would word the issue. If an attacker can talk to the Erlang distribution port with nothing but the cookie preventing them, then the system is already lost. That should never happen, the distribution ports should only be open on an isolated internal network for the other cluster members.

If RabbitMQ opens these ports by default and does not document the issue clearly, then that is the failing. I don't think fixing whatever random number generator is generating the cookie will make the system secure enough to be considered fixed. At worst it may give a false sense of security when in reality the fix has been made in the wrong location.

> false sense of security

You don’t think a random generator that looks like it outputs log₂(26²⁰) ≈ 94 bits of entropy, but is limited by the weak PRNG state space to 36 bits, and further limited by poor seeding to about 20 bits, creates a false sense of security?

(Source: https://github.com/gteissier/erl-matter)

It would be entirely possible to generate a cookie that gives true security. It would also be possible to generate no cookie at all and force the administrator to become aware of the issue if they want to enable clustering. It would also be possible to limit the exposure to localhost only by default.

But Erlang does none of these things. It generates a weak cookie that looks like a strong cookie, leaves it in a hidden file that the administrator may never even become aware of, and exposes a daemon that relies on it for security to the internet by default.

This is not how you build a secure system. This is not even how you build a system to get the administrator to realize that it needs to be secured. This goes against every security best practice that’s been written and some that are so obvious they shouldn’t need to be written. This is irresponsible and inexcusable in today’s environment, and hardly even excusable in the environment that Erlang was originally written for. This needs to be treated as the serious vulnerability that it is, and needs to be fixed.

> You don’t think a random generator that looks like it outputs log₂(26²⁰) ≈ 94 bits of entropy, but is limited by the weak PRNG state space to 36 bits, and further limited by poor seeding to about 20 bits, creates a false sense of security?

I don't think so, as the Erlang documentation states that 1) the distribution protocol is in plaintext, 2) the cookie is not secure in the cryptographic sense, but in the sense that it will prevent accidents, and 3) the cookie handshake is not cryptographically secure.

It is not possible to generate a cookie that provides any true security as the distribution protocol is plaintext and the cookie handshake is not secure. Claiming to get any security benefit from a "stronger" cookie would only give a false idea to the user.

Erlang does not start distribution by default. This is done by the user of Erlang, or the application that runs on Erlang. It is up to them to make sure that their environment is secure and that they understand how Erlang distribution works. Namely, to not open ports in the firewall willy nilly and only allow communication to them from an isolated network or over TLS. Frankly I don't know what you would expect Erlang to do here, maybe to add even bigger warnings to the documentation?

The problem here is application developers not understanding how to configure their application to be secure by default. They are the ones opening the distribution wide to the Internet and not advising their users on how to do it safely, not Erlang.

Reply on news.ycombinator.com