Hacker News Reader

Comment by technion | Hacker News Reader

We've got some information on the timeline (and a name) on one of the major exploits here:

Some of the detail on where this is a mess -

The relevant security update is only offered for the latest (-1) Cumulative Update for Exchange. So you can open Windows Update and it will say "fully updated and secured", but you're not. Complicating matters, Cumulative Updates for Exchange 2019 have to be done from the licensing portal, with a valid logon.

So maybe you have a perfectly capable 24x7 tech team, but the guy who manages license acquisition is on leave today. This is how you may basically find yourself resorting to piracy to get this patched.

View on HN

Reminiscent of Cisco IOS patches being stuck behind support contracts - and inaccessible to many until they pony up.

It's been a while since I've had to deal with Cisco IOS, but IIRC they were always good about releasing security fixes to anyone upon a TAC request.

For used devices off support contract, security incidents were a great opportunity to get free updates.

IIRC you needed a contract login to open TAC cases.

I always just got copies of the .bins from friends who worked at places that had contracts. They didn't gate updates at that time by which model you bought, once you had access you could get firmware for anything Cisco.

Phone or email would work too, no contract required.

> As a special customer service, and to improve the overall security of the Internet, Cisco may offer customers free software updates to address high-severity security problems. The decision to provide free software updates is made on a case-by-case basis. Refer to the Cisco security publication for details. Free software updates will typically be limited to Critical and High severity Cisco Security Advisories.

> If Cisco has offered a free software update to address a specific issue, noncontract customers who are eligible for the update may obtain it by contacting the Cisco TAC using any of the means described in the General Security-Related Queries section of this document.

https://tools.cisco.com/security/center/resources/security_v...

Can confirm this works, did this 3 times in the past. 2 times no questions asked, a 3rd time I first got a message that I didn't have a support contract, but got it after linking them the release notes of the firmware in question where it said Cisco would provide it for free.

Cisco support at Enterprise level expensive or not. The Best in industry, their support organization is literally moving to GCP because they know what to do. GCP don't know yet.

I can confirm that on personal devices with no support contract. You contacted them asking for an update image due to published vulnerabilities and they sent it over

> the guy who manages license acquisition is on leave today

At that point, if you really have no other options, you pull the network plug. Or firewall it to internal-only. Email can wait for a day. And the nice thing about the protocol is that it will all get re-sent automatically.

This depends on the sender's mailserver to cache the mail for a day (or a full weekend) without rejecting it. Some mailservers will kickback mail much sooner.

Such servers are then not compliant with the standard of 4-5 days. See RFC 5321 sec 4.5.4.1.

Are non-standard retry intervals actually that common?

Try telling the whole company email can wait a day. Good luck!

this! i cannot fathom any executive choosing the shutdown of email sevices over some risk that something might happen.

Haven't there been incarcerations due to gross negligence like that?

No? Name one.

(Managers are very rarely jailed for anything except the most deliberate fraud; even negligence that gets people killed is routinely not punished at all)

I don't know any which is why I am asking. It seems like there should be well-known victims of being made an example of.

Why? We don't usually hold victims accountable for crime, and hardly anyone understands computer crime anyway.

There have been a couple of big GDPR fines for customer data breach, but obviously those are made against the company and not individuals.

Was Aaron Swartz the victim or the perpetrator? They set out to make an example out of him. They psychologically tortured him until his heart broke and he killed himself. They were extremely effective in terrorizing us into submission and now no one dares confront our masters again, lest you end up like poor Aaron. No one even speaks his name and we all know his story. And this was only an academic institution so imagine what these completely inhuman actors behind tax heavens and so on do to suppress those they prey upon.

It's a punishment system, not a justice system.

How is this relevant?

It replies to this: "Why? We don't usually hold victims accountable for crime, and hardly anyone understands computer crime anyway."

Is there a specific thing you want me to clarify?

Aaron was accused of a variety of charges, because he'd committed the “crime” of intending to violate corporate copyright on other people's academic papers – but the law was blatantly immoral and the prosecution even more so.

He was not a victim of cybercrime; he was the “perpetrator” of something that was retroactively classed as cybercrime. I know you're upset, and angry – we all are – but that just isn't relevant to this discussion.

I said: "It seems like there should be well-known victims of being made an example of."

"Victim of being made an example of."

I never said he was the victim of a cybercrime. And it IS relevant because because it makes it unavoidably clear what I wrote that someone didn't read with proper care: The financial sector doesn't have this experience. - They don't have their own Aaron Swartz.

Ah. I thought that was a case of “I a word”, where the missed word was cybercrime.

Negligence results in civil trials, not criminal penalties (unless you manage email for the DoD or other government entities).

Huh, you mean literally all email will be exfiltrated to some chinese actor? I’m fairly confident most will find that unacceptable.

you and i might. but for most businesses the prospect of loosing money/contracts because an email was read by noone is probably much more real than loosing because too many people read it.

But exchange is much more than e-mail, is it not?

Calendar and contacts

Shouldn’t you have, or should I say don’t most orgs have, a spam filter or some other GW in front of Exchange that actually accepts the mail publicly? And then that gateway will send internally to the actual Exchange? This is what I’ve seen in a few orgs.

I don't think email proxies are built to cache entire org ail messages that long.

They don't follow the rfc if they don't. Email protocol is resilient to servers not being teachable for a few days. Humans forgot that though and came to think that email is like slack

Okay, I see what you and parent say, and I might be misunderstanding the setup of common email proxies.

Huh, this appears to be a change they made for 2019... the downloads for 2016 CUs, including the latest ones, are available publicly: https://www.microsoft.com/en-us/download/details.aspx?id=102...

Yeah this is not the case for 2013/2016, although Exchange CUs are a full installer you can run a fresh install from. Unusual for Microsoft software updates in that respect I believe, and it kind of makes sense that would require a valid license to download.

Clearly is is less customer friendly than 2016, but then Microsoft do REALLY want that sweet reoccurring subscription for Office 365 (or is it Microsoft 365 now?). Can't make it too easy to host your own Exchange server these days...

> So maybe you have a perfectly capable 24x7 tech team,

> but the guy who manages license acquisition is on leave today.

Then I wouldn't have "the" guy for anything.

It is not the guy that's guilty, but you for having such bad organization that one guy under the buss is sinking the entire ship.

I think that's what they're saying - that the problem is that there is a single "the guy".

Then it should've been "wouldn't have a guy for anything". The issue with funny quips is that if you don't do them correctly ambiguity reigns supreme.

I get what they're saying, but in the majority of organisations the persons who spends their entire day reading Adobe's EULA and counting Oracle client installations isn't considered the sort of role that anyone has ever considered needing redundancy or full time availability until this thing broke out.

I’m pretty sure he’s saying that guy is the problem for not being available and should be fired. The way it’s worded, it’s not clear who has the correct interpretation.

I thought it was Microsoft that was guilty for hiding a security update in the licensing portal!

MS sucks, but a manager can't change them. What you can do is to have good processes, knowledge sharing, and maybe even the balls to take down the entire mail service if it is exposing it to a hack and you need time to patch.

Why do people still use MS products in 2021. Where are developers coming from that actively want to learn which hoops to jump through to be allowed the privilege to briefly use the hardware they rent out to Microsoft?

You mean sysadmins, not developers right? :)

"You took a vacation the same day that a zero day dropped. You're fired"

Yeah uh, I don't think I wanna work for you then

I think their point is that "the guy for a specific task" can't exist on a team that actually is "24x7 perfectly capable".

If your organisation has key person dependencies that’s a problem in itself.

Not all organizations can afford to double up in everything.

That's fine, but that means you don't have a perfectly capable 24/7 tech team.

Yes it does. 24/7 means all the time for any reasonable given scenario. If someone taking leave means you can’t fulfil a function properly then you’re not 24/7 even if you are perfectly capable.

Exactly.

If you cannot afford to double something mission critical up then you at least should have a backup plan what to do in case something critical happens while that one person is unavailable. If you cannot come up with a plan, consider if you should be running this.

Since when are licenses mission critical? ( OP said the update is only available via the licensing portal)

If your 3rd party license expires and takes down your whole system then it is mission critical.

Since you need the licensing portal for critical updates at the latest.

Agreed, but I’m curious how often you see how often this actually works out as it should.

Pretty sure the poster is talking about having one single point of failure for all license acquisition in the first place, not about firing the single point of failure.

You are right. I think the people downvoting you just misunderstood the point you were making: In a “perfectly capable 24/7 tech team”, you should not depend on a single individual for anything.

In real life, however, this kind of thing happens all the time. Someone forgot to write down the login when they left and no-one caught it in the offboarding. Or someone set up 2FA on a system but didn't put that info into 1Password / the wiki, etc.

Unfortunately a few comments here have honed in on one contrived example of why I think this strategy is broken. To give another contrived example: I personally had a logon to this portal, but it broke last year when they integrated logons with Azure and it took me like three months to get it fixed.

The fact a critical security update can't just be downloaded is bad. I don't care if someone in sales thinks every licensed user should probably be able to get it. Here NCC produced a list of "valid" files to help people scan for not legit files. Except they don't have Exchange 2019 CU 8 because they couldn't get it:

https://github.com/nccgroup/Cyber-Defence/tree/master/Intell...

Microsoft has a hard limit (5?) on the number of individual accounts you can grant access and in a big enough org it's still plausible they'll be scattered across the world and you'll find none of them available the exact hour you need this update.

The failure mode of clever? It's "asshole." -- John Scalzi

I know you’re trying to save the original comment but that comment can legitimately be taken the way the downvotes are taking ... that the commenter believes that guy should be fired for being away from his phone. Why legitimately? Because I’ve worked with people like that.

Reply on news.ycombinator.com