Hacker News Reader

I see bug bounty programs as just being another example of InfoSec charlatanism. People do bug bounties because InfoSec people say you should do bug bounties, and not doing what InfoSec people say you should do implies some sort of malfeasance. But the value of them is remarkably questionable. For starters they are absolutely not a replacement for pen testing, having a bug bounty is not a sufficient substitute for any of the pen testing that you should be doing. The value of the bugs you have reported are also typically incredibly low value. Most of the reports you get through bug bounty programs are just the output of open source scanning and static analysis tools. You get no-effort reports for things like frame-able content and “your mobile app has a dependency, which has a dependency, which was compiled with marginally sub-optimal flags”. Actually valuable reports do make it through, but I seriously doubt having a bug bounty program is more effective than publishing a security email address on your website. They’re mostly just spam generating services that invite people to try pressure you into coughing up some money for largely trivial nonsense.

Bounty submissions are mostly scanner spam; maybe almost entirely. But every once in awhile you get a serious and important report, and you wouldn't have gotten it without the bounty. Retain 3 different pentesting firms to hit an app and their reports will overlap maybe 70-80%; similarly, bounty people find different stuff as well.

> But every once in awhile you get a serious and important report

Yes this is true.

> and you wouldn't have gotten it without the bounty

Of this I am highly skeptical. Most companies that I’ve worked at who don’t have bug bounty programs get the occasional serious report. These reports were submitted to companies before bug bounty programs were even thought up. Bug bounty services just seek to capture value from something that was happening long before they were invented, and I’m highly skeptical of them having a significant impact on generating serious reports that otherwise wouldn’t exist.

The other issue is that everybody has a finite security budget, and dealing with the spam is going to (perhaps significantly) eat into that. In those cases, it’s taking money that could be spent on something with a decent ROI, and redirecting it to essentially just creating busy work for your security team (or spending their budget on paying somebody else to do that busy work).

I think they really only exist because a lot of security professionals don’t really know what they’re doing, and a lot of their employers don’t really know what they’re supposed to be doing.

I watched it happen, several times, on projects where I not only had 6 weeks of undivided time to run an app assessment before I took over their bounty program, but the client had contracted assessments from other firms prior to me being involved.

It's also a refrain I got from almost everyone I asked about running bounty programs.

A fundamental truth I don't think anybody who does appsec assessments professionally can escape: multiple audits of the same target will turn up different bugs. It happens even if the same people do both assessments! I've never talked to a software security professional who pushed back on this; maybe you'll be the first.

> A fundamental truth I don't think anybody who does appsec assessments professionally can escape: multiple audits of the same target will turn up different bugs.

I completely agree with this.

> I watched it happen, several times

I don’t doubt you have, but this is anecdata. I wouldn’t want to draw any conclusions from this, especially because it conflicts with a lot of my own anecdata.

Auditing/security testing is something you could infinitely devote incrementally more budget to, and infinitely receive incremental returns from. While I question it provides as much benefit as you’re describing, it can certainly provide a benefit. My issue with it is that, in my experience, it has always had a bad RoI. The resources that you have to devote to dealing with the spam make extracting any value from bounty programs a very expensive exercise. I’ve personally seen people devoting more than one day a week to managing bounty programs that they were lucky to get one good report per year out of. Imagine how much more value they could have gotten from having a security engineer spending a full day per week threat modeling with the product engineers.

Perhaps the ROI is different if your department is over-funded, or if you have a big brand that people want to write self-promotion blogs about. But any time I hear a security professional parroting the importance of bug bounty programs, the cynic in my wonders whether they’re just looking for some low-skill busy work for themselves.

I think we'd have to struggle to find a clear debate between the two of us. I just wanted to make the point that people run bounty programs because they do in fact turn up black swan bugs, even after the repeated pentests that most of these firms have already engaged.

There are several other important price inputs to this market, including scanning services (thousands quarterly) and consulting penetration testing (tens of thousands annually). Most companies that offer bounties have also had pentests done; nobody believes that pentesters are underpaid (though maybe they should).

The reality is probably that 80% of the vulnerabilities disclosed on bounty platforms just aren't worth that much. Certainly: companies that lack security expertise but manage bounties tend to radically overpay bounties; I'd of course be curious to see a breakdown of that $100MM by bug class.

> Companies that lack security expertise but manage bounties tend to radically overpay bounties

Could you share your reasoning on this? Are you saying they're overpaying for a low threat type of vulnerability, have a large number of vulns that should have been caught earlier, or something different? Genuinely curious! Thanks

I'm saying that people are submitting sev:info's and getting paid $500 for them.

Makes sense, thanks for the clarification.

"reducing that by selling information to a different broker, who will eventually find someone to weaponize or monetize a piece of the exploit" This has become the standard way of monetisation for the more capable ones who don't want or don't see the point to do the crime itself. ATM skimming kits, carjacking kits, etc. The problem is that security is not rewarded at all.Even a top level expert with very deep knowledge gets peanuts in comparison to the cost businesses incur when they get hacked.

Obviously there's a way to make the market more efficient, let black hats compete to purchase exploits on the same platforms .... of course that's likely illegal, for good reason. Instead we have two markets with only a few people playing in both so price signals are likely weak

I don't think it's entirely fair to say there is no competition in this market. In the end companies are still competing for a limited supply of ethical hackers. Except that it is a bit of an unusual market, since both sides are operating with very limited information.

Perhaps it will become more stable when people realise you can sell negative results as well (either implicitly by cooperating, or through an actual market). Anything that can bridge the gap between hackers wanting to get paid for their efforts and companies wanting to pay for results would help.

More volume would also help, in the end it's hard to sustain a whole industry on just tens or hundreds of millions per year.

> I don't think it's entirely fair to say there is no competition in this market

Good thing I never said that. The market is inefficient. I like that idea of monetizing negative results. More transaction volume generally helps, I think external market signals and their actual value are whats really missing.

Like, how much would someone pay for an exploit should dictate how much a company should pay for letting a crowd try to fix it where only one person/group gets the payment. If there was a way for additional participants to get paid for bothering to look at all that would be helpful too, and different people can focus on how people try to game that instead of worrying about the idea of people gaming it.

Aren't negative results already monetized, through standard pen-testing servcies? The usefulness of a negative result is dependent on the thoroughness and skill of attacker, while a positive result is useful no matter who submits it.

An unknown person finding a vulnerability has clear value, but a negative result from an unknown person doesn't mean much.

A negative result has value as long as it's clear what vulnerabilities have been tested. Of course it's pointless if you don't trust the person testing it, but that's always going to be a problem. I guess you might need to have reproducibility to make it truly useful.

That seems to me to be mostly limited to automated tools, or checklist level evaluation?

The good/dangerous bugs require creativity and deep system knowledge to find, which is hard to trust that an unknown person has.

Then comes risk management and I would tell you "good bugs that require creativity and deep system knowledge to find" are not dangerous.

It is sexy and all to sell "we hacked you using emacs through sendmail by the moon phase exploit that happens every 2 months". But really how many people will be able to exploit that? My company is not target of a state sponsored actors. Automated tools, checklist evaluation and decent administrators will be good enough for most of companies. Ordering pentests from time to time is good practice to check if administrators are doing their job as in "trust but verify".

It's not. Running scanners and report scan results won't get you paid so it's a complete waste of time.

Trust is easy. You either have a proof of concept and can show impact in your report or gtfo basically.