Zero-Day Exploitation of Atlassian Confluence | Hacker News Reader

Zero-Day Exploitation of Atlassian Confluence

volexity.com

347 pointsafrcnc112 comments

Both Atlassian and CISA are recommending either disconnecting servers from the internet or shutting them off entirely. I don't like the sounds of this.

https://confluence.atlassian.com/doc/confluence-security-adv... https://twitter.com/USCERT_gov/status/1532511428451631108?t=...

https://confluence.atlassian.com/doc/confluence-security-adv... is updated with a workaround that you can apply manually now.

Posting this high here so it can be seen.

Hacked together a quick and dirty shell script to install on my boxes in a repeatable way... Hope this helps others that find themselves in this situation.

https://gist.github.com/noahbailey/0260efa836b4ea67163cbfeef...

And now the same link has the released update versions. And more details: https://jira.atlassian.com/browse/CONFSERVER-79000

Part of the fix is a patched version of xwork 1.0.3, originally released in 2005, which has had several CVEs past 1.0.3. Yikes.

Which is very strange, they usually have at least a workaround or URL to block.

There are now manual steps for mitigating the vulnerability.

See https://confluence.atlassian.com/doc/confluence-security-adv...

It is strange that they didn't have mitigation steps earlier, but I'm guessing Atlassian announced this immediately since it was already being exploited.

They’ve suggested blocking ${ in the request

Using nginx in front of my local confluence server (not listening on the internet) and used the following:

  location ~\* \$\{. {
          deny all;
  }

but this only helps if this vulnerability is triggered using the request uri otherwise it's useless

Anyone know how you'd quote it for Apache rewrite?

        RewriteEngine On
        RewriteCond %{REQUEST_URI} ^.*\${.*$
        RewriteRule ^/(.*) https://google.com [R=302,L]

seems to work ...

The security advisory from Atlassian says blocking ${ "may reduce your risk". That's a far cry from being a fix.

I really worry about these sort of claims. I'm seeing pushback from people about shutting servers down because it's insisted "a WAF is protecting us now", and that's based on a "may reduce risk".

I would worry about just using that blocking pattern. They mention OGNL injection and ${ is just one pattern you can go after. There's also #{, %{, #var, and more.

Same issue with log4shell. "Why are you bugging us? We already applied <mitigation posted 2 weeks ago that has since been disproven>."

posting a snippet for HAProxy in case its helpful to someone else (or if others have recommendations on how to better do this)

  http-request deny if { path -m sub ${ }
  http-request deny if { query -m sub ${ }
  http-request deny if { path -m sub $%7B }
  http-request deny if { query -m sub $%7B }
  http-request deny if { path -m sub %24%7B }
  http-request deny if { query -m sub %24%7B }

Sorry what now? You cannot simply block the current attackers control addresses (they actually do give you these) or domains. First they could be either VPN users or "residential proxies" aka legitimate people with a botnet on their pc. Second the attacker - or anyone else who now knows it exists - can just change URLs or IPs. They can change the exploit signature so anti-virus and IDS systems can't trivially see it.

The ONLY correct solution here is to bring the servers offline until there is a patched version to upgrade to. Anything else would be a terrible idea.

_sometimes_ there is only one config setting that's affected, or some other often lesser-used feature that can be disabled. But it highly depends on the method used.

GP meant that oftentimes exploits use a specific entry point on the public surface and blocking access to it acts as a stop-gap measure.

We’ve applied protection for all Cloudflare customers, including those on our free plan: https://blog.cloudflare.com/cloudflare-customers-are-protect...

It's funny how over the years I've come to both love and hate (insert Michael Kiwanuka joke here) Cloudflare more and more. The number of use cases where there is substantial benefit is now much larger than I ever foresaw when you guys launched, at the same time Cloudflare now has so much power that any mistakes or changes in direction that bother me have far more impact than before.

Best of luck in navigating this, it's a very fine line.

(For the record, Cloudflare, Stripe and a limited set of other companies are in my opinion some of the few that 'made it big' and that didn't turn into disasters or abusers of their position, in contrast with Google, Microsoft, Apple, Facebook and many many others)

This is the paradox of centralization, isn't it -- we can't run our own email servers anymore, because we can't send email to the major providers. We can't run our websites without engaging with CAs because HTTPS is a requirement. We won't be able to run our own platforms, because without the protection of a well-resourced middleman like Cloudflare to stay up 24/7 mitigating our issues, the overcomplexified and permanently insecure state of modern software would be intolerable.

To expand on the irony: All of the things you mentioned are deemed necessary, because the Internet is inherently open and decentralized. Email needs protection against spam and phishing, HTTPS protects against man in the middle attacks and eavesdropping, Cloudflare have services that protect against DDoS attacks and to provide read scale among other things.

I don't think the Internet was envisioned as a cyber war zone.

> I don't think the Internet was envisioned as a cyber war zone.

It was seen as a communications tool during a war, not as the battleground itself...

Indeed it is. And I think this is the reason the centralization/decentralization pendulum is there in the first place. But in the past the return (decentralization) of that pendulum was driven by major advances in technology, this time around that won't be happening and absent another driver we may be more or less stuck.

>we can't run our own email servers anymore, because we can't send email to the major providers. We can't run our websites without engaging with CAs because HTTPS is a

I call it BS, i made many many emailservers, and my customers have no problem's with sending to gmail/hotmail etc.

You either got incredibly lucky, or are talking about experiences in the (relatively distant) past. I say this as someone who also used to run email servers, and long since gave up on it.

Or maybe you don't do it right. SPF (with p=reject), DMARK, DKIM, correct reverse DNS entry, Certs. And for google and hotmail a site-verification code <- This is what you have NOT done right?

Or maybe it's really just luck for the +50 Email-Servers, who knows right?

In response to your final paragraph, I would note that reversion to the mean is a painful, painful thing. It's only a matter of time.

In response to that I would like to note that fish rots from the head. Have a look at the kind of people that run each of these companies and you'll find some remarkable differences.

Today, yes. At some point those people will rotate out.

Well, given that eventually the Himalayas will be flat ground again, yes, sure. But for now those stand out and that sort of reasoning can be used to negate anything in the present. If anything I'd hope that it would inspire them to try really hard not to revert to the mean.

I don't think it negates the present, really. More a matter of making sure we recognise what we've got while we've got it.

If I look at those other companies roughly the same time after they were funded then I think that the only one that deserved the benefit of the doubt was Apple, and only because of Woz's influence. Which gives me some hope.

I completely agree. It's one of those fantastic things that can turn into a nightmare in the blink of an eye.

This is a great case for why proxy pre-auth is so important. Doesn't matter how buggy and riddled with worms your app server is if nobody can get to it without a valid token. Sure, it introduces plenty of other problems, but it does really help with this. At the end of the day, you shouldn't be able to execute anything until you're authenticated.

Probably would have helped a bit if their java server process wasn't running as root as well.

No one sane runs confluence as root. But RCE is RCE. It's always serious, even if the user isn't privileged. It's significantly easier to get root access once you have access to an unprivileged acccount than without. And with tools like Confluence, there's a good chance that if you control the application, you can intercept login requests and steal credentials. Which can then likely help you gain access to other services.

It would help if the run-user was not the same user that owns the files on disk. The run-user should only need to write logs and caches and databases. Not jsp files.

I usually try and make this so, but it is often very hard to impossible. Vendors do not make it easy.

>No one sane runs confluence as root.

See "Figure 1. Process tree in Volexity Volcano Server from the infected Confluence server"

https://www.volexity.com/wp-content/uploads/2022/06/Figure1....

I didn't mean running as non-root would fix the problem, just that a remote root RCE was worse than a normal RCE.

> And with tools like Confluence, there's a good chance that if you control the application, you can intercept login requests and steal credentials.

I thought Atlassian products were well trusted. Is that not the case?

They've had a string of script-injection vulnerabilities with similar severity -- the last was less than a year ago:

https://confluence.atlassian.com/doc/confluence-security-adv...

Each time, they patch the particular vulnerable script, but they don't seem to have done anything systematic which would eliminate the entire class of vulnerabilities (like, say, adding taint-checking for unquoted, client-supplied content before running code as an OGNL script). So they keep coming up.

They’re widely used, which is not the same. If you’re a customer you’ll routinely encounter bugs, clumsy or inaccessible UI, annoying limitations, and performance issues which take ages to get fixed if ever. I would not assume that they’re better at security than everything else.

Nailed it.

Let's also admit that setting up iptables is harder than it needs to be. I mean, not after the first twenty times... you eventually remember all the arcane logic and expect everyone else to as well.

If you're manually editing iptables rules in production in current year you're doing it wrong anyway, you should have some form of config management like ansible/salt/puppet etc that make it dead simple to add rules or at the very least just use firewalld or something

In the past, I've found making tomcat listen on unprivileged 8080 and proxying it using nginx on 443 works well.

Even better running the system as a Docker, or other, container image with read only filesystem.

who in the hell does this?

Mostly Linux-illiterate people, who want to bind to privileged ports (443), and have no idea about systemd sockets[1] and socket-activation[2], so they stick with the first solution that works.

Most people don't know that you can have systemd bind to the privileged ports for you, and hand over the fd to the port to your (unprivileged) process.

Same goes for docker/containerd btw, if you run docker as root nowadays, you are doing it wrong.

[1] https://www.freedesktop.org/software/systemd/man/systemd.soc... [2] https://www.freedesktop.org/software/systemd/man/systemd-soc...

False sense of security or just plugging one hole in a wedge of swiss cheese. It isn't always the server, sometimes it is just the framework. A proxy won't stop that.

https://www.zdnet.com/article/equifax-confirms-apache-struts...

The metaphor of plugging holes in a wedge of swiss cheese is actually the basis for the aviation industry's security model!

https://en.wikipedia.org/wiki/Swiss_cheese_model

It's more a risk management model than a security model. It tries to build a safe system from layers that are not 100% safe (e.g. because components on a plane may fail or people may not follow procedures exactly). That's why in an airplane accident it's not usually one single cause, but a whole string of things that have gone wrong. But still, we should try to keep the holes in the individual layers of cheese as small and as few as possible...

This is really neat and novel to me, thanks for sharing.

The IT version of it is sometimes called "defense in depth" - even if each individual layer has weaknesses, combining them gets something stronger.

This often doesn't apply in cryptography; where adding two separate functions may result in a system that is actually weaker to attack.

Well yeah, if you put two slices of Swiss cheese side by side instead of on top of each other, you have given the attacker a wider choice of holes...

Your link points to a vulnerability that could have been mitigated by putting the internal application developped with Apache Struts behind a proxy server like Nginx or GCP Identity Aware Proxy (IAP) that authenticate the request before the request is forwarded to the application server.

This only applies to apps for internal use like Confluence or Jenkins, not publically accessible websites.

Proxy preauth could be able to mitigate a vulnerable framework. The idea is that you never connect to the application server itself, but to a proxy that requires a separate authentication. That’s obviously not possible for a public-facing application.

Now, if the attacker has access to the proxy authentication, all of this doesn’t help much, but at least you could track and limit who can access what chunks.

My confluence server is behind a proxy which requires a valid client certificate to talk to the server

I have other proxies which require OIDC authentication before passing any packets on

Now sure, I'm still vulnerable to internal attackers from my company, but that's an audited trail to a real specific user, not just a random botnet on the internet.

Smaller attack surface internally, for sure. I also assume internal networks are hostile, not just from internal threats, but from external lateral movement.

I'm don't follow. If you have a pre-auth reverse proxy in front of your web application, you drastically reduce your attack vector.

An unauthenticated attacker can no longer simply rely on a vulnerability in the application or underlying framework, but must first obtain an employee's credentials, or find another vulnerability in the proxy or authentication infrastructure.

This isn't a false sense of security. It is a concrete additional layer of security which has tangible benefits.

I don't * - Too late to edit.

Plug enough holes though and at some point it starts to have effect.

The real problem is if there only one wedge, you need to combine that strategy with multiple layers of security to get anywhere, otherwise the first hole is the last.

Exactly, I mean everytime I saw any of this self-hosted SW, open directly to internet, my eye is starting to tick.

Agreed, if not a proxy then put your services behind a VPN or a bastion host when possible.

And the likes of Cloudflare access or VPN only access via Tailscale make it so easy to keep it protected. I'm always a bit shocked when I come across publicl ogins for Jira or Confluence instances of large corps.

Yup. Really easy to set this up with the IDP's available today too.

In the IdP/SP/SAML sense of it, I'm really used to that level of validation actually hitting the application and being processed through an authentication subsystem, rather than being done at the webserver level.

Possibly just some nginx features I'm not familiar with, or are there whole apps out there that serve to be nothing other than an authentication layer inside of a webserver? Are they popular?

Sorry if I misunderstood the question, sounds like you are asking if there are some common tools that do auth outside of the application before passing traffic back.

Google IAP is a big one, and Cloudflare Access can let you do similar things (link an internal service to Cloudflare network and Cloudflare will deny someone ability to talk to that service until they've validated their identity to cloudflare and it matches your rules somehow.

Azure AD Application Proxy fits the bill for Microsoft 365 shops.

Nothing from Atlassian should ever be exposed to the public internet. Atlassian stuff is a security tire-fire with very old dependencies even in the latest releases, a painful reinstall/config merge needed for even minor patches, and no auto-update mechanism.

You absolutely do, but I was more interested in the mechanics of it at the webserver level rather than SaaS equivalents. The Google IAP one is helpful as I only knew about Cloudflare Access.

HTTP Redirecting out to an idp and it posting back to the application with a SAML assertion is pretty common, if that's what you mean. Lots of people do that with things like O365, the AWS Web Console, etc.

Of course. Just asking how is the web router aware of whether or not the redirect is necessary if you're not hitting the application to validate login state.

A quick update -- we've just notified Confluence Server and Data Center customers that we expect security fixes for supported versions of Confluence will begin to be available for customer download within 24 hours.

We will continue to update our advisory (https://confluence.atlassian.com/doc/confluence-security-adv...) at least every 24 hours as additional details become available - including a download link to the software updates as soon as they are available.

I wish these emergency patches could be made available as a marketplace app to install rather than a complete code update; as the latter takes much longer to implement.

Hey! This temp fix isn't a bad one, it's easy to apply and restart.

https://confluence.atlassian.com/doc/confluence-security-adv...

From: https://confluence.atlassian.com/doc/confluence-security-adv...

"If you are unable to take the above actions implementing a WAF (Web Application Firewall) rule which blocks URLs containing ${ may reduce your risk."

Smells like log4j issues?

Edit: Ah, template injection, see below.

It's a common pattern in Java injection vulnerabilities. You can see an example here:

https://portswigger.net/research/server-side-template-inject...

Part of me believes that the lack of workaround or patch and the vagueness of the warning, coupled with their cloud product being safe is a very happy accident.

Given how much they’re pushing the hosted offering.

> Atlassian Cloud sites are protected

> If your Confluence site is accessed via an atlassian.net domain, it is hosted by Atlassian and is not vulnerable. Our investigations have not found any evidence of exploitation of Atlassian Cloud

Must be a relief for Atlassian after the recent long outage that only impacted cloud instances.

Which is strange because it means either something catches the exploit or it’s particular to the stand-alone code, as cloud is mostly (but not entirely) DataCenter.

They also might have a hot fix (or temporary feature disable) that they've applied in Cloud, but not yet packaged for distribution.

Maybe just a WAF rule like Cloudflare rolled out. But then they should've stated that since there is a chance cloud instances were breached.

Their communication around the recent downtime was pretty terrible, so I wouldn't be surprised if they just didn't mention this and took it as a fact. Then again, they may have confirmed it via logs.

Their cloud and on-prem offerings diverged quite a while ago, so it makes sense

Great timing. Here in the UK it's a four-day weekend. Can imagine that many affected will see this too late...

Not monitoring the components in your setup for security announcements is a fairly basic error. Someone should be watching these. Otherwise every Christmas would be a hackfest.

> Otherwise every Christmas would be a hackfest.

It probably is.

I raised the alarm for log4shell internally on December 9th, and then then it was being actively exploited. I know people at other companies who hadn't heard about it, or didn't think it was worth doing anything about, as recently as April.

That’s where diversity hiring shines.

https://confluence.atlassian.com/doc/confluence-security-adv... is updated with a workaround that you can apply manually now.

And now the same link has the released update versions.

Sidetrack, what's a good collaborative wiki? I've tried a bunch but always found Confluence much more polished than other offerings.

Have a look at https://www.bookstackapp.com/. It's open source, very polished and also useable for non-techies.

I'm working at XWiki SAS [1], which provides hosting, support and customization for XWiki [2], an open source extensible wiki you can use for collaboration in a team (among other things). We are seeing many clients coming from Confluence because of the recent price raise and so we have tools to migrate from Confluence. They seem happy. There are ways to customize XWiki to fulfill the specific need of each team if the wiki is a model that suits you. There are a lot of extensions available and you can program in the wiki itself to adapt it if standard XWiki lack some specific feature you'd need.

I don't know Confluence though, so I cannot say how more or less polished XWiki is compared to Confluence.

Anyway, don't hesitate to reach the XWiki community (there's a public forum and a public matrix channel, come say hello!) or XWiki SAS if you have questions or concerns, they'll be happy to answer and you can see whether XWiki would be a good fit for you :-)

[1] https://xwiki.com/

[2] https://xwiki.org/

When I evaluated some last year, my favourites have been Bookstack and Wiki.js. Both are open source.

A not so good one is Onenote

We've seen a _lot_ of folks coming across from Confluence to Outline since Atlassian stopped offering on-premise licenses. We have a cloud, self-hosted community, and on-premise offerings.

https://getoutline.com

After a couple of hours of customization, dokuwiki is excellent.

I've found it pretty usable out of box, though I guess it depends on exactly what customizations you normally recommend.

Pick a theme that meets user expectations; add plug-ins that make sense; set up authentication that fits your use case; create a site skeleton that fits the way people will use the site. Things like that.

The security advisory has been updated with new information regarding a fix for for Confluence Data Center and Server products. Please see the advisory for more information and updated instructions.

https://confluence.atlassian.com/doc/confluence-security-adv...

Not long after log4shell yet another example of why JEP 411 (removal of SecurityManager) is a very questionable decision.

Running applications under SecurityManager would have prevented a lot of these vulnerabilities (or rather - their severity would be much lower).

Why? You still get full access to everything the app can see - all Confluence content. To prevent access to the rest of the system, there are better ways now (Containers). YMMV, but SecurityManager was never really usable in most real word use cases.

> You still get full access to everything the app can see - all Confluence content.

No (with proper policy in place). That's the whole point of SecurityManager. Even if you can execute code - it is sandboxed and has limited (or none at all) permissions.

To be fair, with most stuff running as VMs, a confluence server will usually not run anything else. So root doesn't really change much here I guess.

VM might have access to more cloud resources via bad AIM policies. Getting access to root even in container can be just as bad.

Why is removing a legacy component which is effectively ignored and unusable by most modern applications an issue? SecurityManager is very fine-grained so you have to spec out every file access by your application (including those of third party code you may or may not know about). Most server side Java deployments run without it so you're better off making sure the permissions of the server process are correct than customising the Java policy.

1) why would you not use a vpn or other gateway between confluence server and the internet?

2) presumably for the same reason they ran confluence as root: no idea what they're doing (forgivable) or lazy (unforgivable)

Also a surprisingly large number of publicly available documentation is hosted on internet-open confluence pages.

Have you looked at the security of VPN products in the past 2ish years? Basically every one of the big ones had several really dumb and easily exploitable RCEs.

It doesn't really help you to use a security product in front of your vulnerable product when the security product turns out to be roughly equally vulnerable.

Awesome. Maybe finally mailbox.org (and lots of other providers) will apprehend that using the same password for Jira helpdesk and my email inbox is a terrible idea. Then again, what privacy can one really expect from email anyway?

This lands the day after di2e.net shutters their IL-2 instance? Coincidence?

TLDR. But can I delete the tasks from the damn thing without logging?

I haven't seen why exploits have been released but this doesn't feel like responsible disclosure to me https://en.m.wikipedia.org/wiki/Coordinated_vulnerability_di...

From what I understand, the vulnerability had already been exploited in the wild. Voxelity discovered it thanks to it being exploited at a client's system.

Reply on news.ycombinator.com