Hacker News Reader

Comment by oglopf | Hacker News Reader

I know for a fact that they also had Splunk ES bought, built, and running, then the CISO had them turn off the alerts because they were "too noisy" and sure enough, they could see all this happening when it was retroactively inspected.

There are so many things wrong in that sentence but my most glaring question is: why on earth was she even receiving the alerts? That's so far below her level. So, there's more here than just the cert issue. There's also just a complete disregard for a secure mindset from the person that is supposed to be setting that mindset in the culture, along with what sounds like micro-managing to a state of no-managing.

View on HN

An Equifax exec went to Parana Bread (of all places) and later had trouble even understanding what a security researcher told them they had an API with customer data that was easily found and queried from the internet.

The executive got confused and suspicious when the security researcher offered to send additional data and talked about sending that data securely via PGP (it seems like he simply didn't know what the request for a PGP key... even meant).

https://arstechnica.com/information-technology/2018/04/paner...

It would seem a lot of Equifax executives are happy to act on their own in spite of their own ignorance.

I know a handful of folks in security, they run into complete morons in the security industry all the time. It's scary how wildly variable the talent in the industry is at all levels.

To some extent that talent variability is everywhere, but the sheer volume of people are so clueless that they could be described as a liability in some situations is surprisingly high when it comes to security. I know security folks who have changed jobs simply because they didn't want to be associated with someone who took over at their employer. Reputation among the folks who know seems to be a big deal to many of them.

Oh I remember this. Dumbass had no idea what kind of issue he had on his hands and just blew it off not realizing the level of blowback and publicity that would occur if the whitehat decided to publicly announce it. Boggles the mind how these people get such high paying jobs.

It is so confusing as the pattern of:

1. Dude contacts you.

2. You fail to work with him / or just don't.

3. It becomes public and you look like idiots.

That pattern is so established that every security executive should know you want to avoid that... but nope, some run headlong into it.

I'm no security dude at all, and I can recognize that pattern just from reading the news. What on earth is an executive reading if they have no idea?

A fair number of people wind up in executive security roles with no technical background to speak of.

That said, for every legitimate security researcher, there are dozens of people who are blowhards. Bug bounty programs, for example, are stuffed full of "[CRITICAL][ACCOUNT TAKEOVER] EMERGENCY VULNERABILITY"... which is actually some skiddie reporting someone being able to XSS themselves.

On the outside, you mostly just see the ones that go bad. On the inside, any given report is unlikely to be nearly as severe as it's claimed to be.

Oh I hear ya. The lack of knowledge extends to the "researchers" for sure.

The unfortunate thing about the Panara Bread deal was any dude could have just pulled up Postman and hit the API and seen the results / issue in a couple seconds + however long it takes Postman to start on their computer.

They didn't even try, they could have rule out "crazy researcher" in seconds.

I do security consulting and that very situation is common. I know several BigCorps that specifically turned off Splunk alerts because it was too noisy. Some of them had valid reasoning for it, other didn't.

There's also no guarantee that if they did have the Splunk alerts enabled, that they would have caught anything. In one instance I can think of, having Splunk alerts enabled did more harm than good because it meant SOC staff were busy chasing down hundreds of false alerts (aka the "noise" in "too noisy") and missed real threats. With the alerts off, the staff could use other more reliable sources of threat notifications and catch more of the real threats.

Unless you have a big enough staff, or have Splunk (and your monitoring team) tuned well enough to actually effectively sort through the alerts, they truly well can just be "too noisy".

> Unless you have a big enough staff, or have Splunk (and your monitoring team) tuned well enough to actually effectively sort through the alerts, they truly well can just be "too noisy".

At some point, the size of your staff doesn't really matter, unless it's at roughly number of alerts / 2. Now not only you have the alerting noise, but you also have the added noise of the big intra-team communication. Cleaning up those alerts needs to be first priority.

Not to mention, usually such a barrage of alerts means they are very poorly designed.

I don’t get it. Can’t you tweak the settings on these alerts until they’re effective never sent during normal operations?

I mean, that might take you a day, maybe a week to catch the edge cases, and then it’s not a problem any more...

Oh yeah, these tools are all trainable

In my experiences though, there’s no time to become an expert anymore

You get it set up, then a PM or manager with no clue redirects us to the next priority. Meanwhile the new monitoring tool just hangs out being underutilized at $20k/mo

Not to say all PMs or managers are clueless, but there are more clueless ones out there than not. They are noise generators themselves.

Or, IMO worse, the sycophant coders who concern themselves with their own personal metrics, release dumpster fire after dumpster fire. And us in devops and secops are fighting their idiocy all day, not training tools to catch intruders.

I’ve worked with far too many people who epitomize that meme of Captain Kirk where he can’t hear you over how awesome he is

Turning off alerts is a quick way to ensure that they are effectively never sent during normal operations.

Most days, it is not incorrect.

> Most days, it is not incorrect

This feels a bit like saying a stopped clock tells the correct time twice a day.

Turning off the alerts doesn't cause a problem unless there was something worth alerting about that day.

And how did you get into this type of consulting?

Depends on how big their team is and its structure with regard to alerting responsibility. One would expect one of the big credit rating agencies to have a large IT team, but old school corporations view IT as a cost center and like to run skeleton crews (which can flatten out administration hierarchies).

I work for one of the big three and we generally are pretty lean when it comes to staffing.

Our CEO likes to claim it benefits us because then they don't have to do big layoffs during downturns.

"Can't chop off heads if there aren't enough to start with" .... now there's MBA logic for you.

Well, if there are zero consequences to failure, why spend money preventing it? Maybe it does benefit companies in that position.

I wrote a lengthy response on how suppressing false alarms is absolutely the right thing to do, then realized that the CISO was the one requesting it.

That really makes no sense. The engineers should be the ones determining which ones to adjust and which to suppress. Middle managers are already too far removed from the productive chain to even make the determination if an alert is false or not, let alone C-level.

> Middle managers are already too far removed from the productive chain to even make the determination if an alert is false or not, let alone C-level.

This is assuming the security team was big enough to have that many layers to create such a disconnect...

It's very likely that she comes from corporate and thinks she has to know everything. I worked for a guy like that. He wanted everything on his plate and simple things like a confluence update could take 1.5 years because of it.

> I know for a fact that they also had Splunk ES bought, built, and running, then the CISO had them turn off the alerts because they were "too noisy" and sure enough, they could see all this happening when it was retroactively inspected.

Splunk (ES or not) is not a magic bullet. It generates a lot of bullshit to justify its expense.

Finding the signal in the noise is easy when you know what the signature of the largest data breach in history was supposed to look like. What exactly does this prove?

> Why on earth was she even receiving the alerts? That's so far below her level.

Knowing the C-levels there, they likely wanted earlier warning of emerging incidents-- EFX and their subsidiaries had been dealing with a steady stream of incidents (some public, some not) for a while before the big one. Krebs covered some of them.

The incredibly frightening thing is that I'm no longer surprised about the fact that people like these still remain completely employable.

Unfortunately, management roles are often inherited, instead of being earned.

< then the CISO had them turn off the alerts

I'm reminded of the Target breach.

Reply on news.ycombinator.com