← Back to context

Comment by SwellJoe

8 years ago

I think I'd need to see some sort of evidence of this assertion. Bug bounties are commonly offered across a huge variety of online services, and they get results...not always, not necessarily consistently high quality, but even the giants (facebook comes to mind) have had reasonably serious bugs found by people seeking bounties.

6 comments

SwellJoe

Reply
dsacco  8 years ago

He's not wrong about the noise level. I conducted a survey of the most notable bug bounties in 2014 and found that the largest companies either have ineffective programs or quickly scale teams to handle inbound reports full-time. There are security engineers at Google and Facebook who spend a majority of their time responding to, and triaging bug bounty submissions.

That said, I disagree that bug bounties don't work for CDNs. You can scale a bug bounty up, it just requires resources. Cloudflare has those resources, and part of it is a function of the reward tiers you offer.

  • kevinr  8 years ago

    Bounty researchers aren't the only quasi-rational economic actors in this sytem. Cloudflare, we might surmise, get enough benefit from their bounty program that they're willing to pay for its administration costs and the occasional T-shirt, but they don't see value in spending more than that.

    More than that, access to the service is actually the limiting factor for good bug bounty results. Cloudflare's bug bounty, we might surmise, works as well as it does because anyone can sign up for a Cloudflare account for free. For an enterprise CDN, who won't talk to a potential customer without the prospect of an $x0,000+/year contract, everyone who has enough access to the service to, in the general course of business, find and submit meaningful reports is employed by a customer, and likely prohibited from accepting substantial rewards. Everyone else either doesn't have enough access to submit meaningful reports, or the bug is so bad (like this one) that they'll report it regardless.

    Arguably this shows that Cloudflare and other CDNs are right in their calculations: Tavis disclosed this bug to Cloudflare without promise of a payout, or even a T-shirt. Might some good Samaritan on the Internet have noticed the bug and reported it earlier if the bounty was more substantial? Perhaps. But in responding to a vulnerability of this magnitude, you want to work with someone of Tavis's caliber, who has the good of all the stakeholders in mind, not a profit-motivated rando.

bsamuels  8 years ago

I'll gladly offer some anecdotal evidence:

We've got about 2500 tickets in our ticketing queue that have been filed over the past 8 months (excluding spam). Out of those 2500 tickets, only five are valid issues, and only one came with an actual write up.

The signal to noise ratio is absolutely awful - and it's not uncommon for people with invalid issues to demand that you pay them regardless.

  • SwellJoe  8 years ago

    Wow, that's much worse than I would have guessed. I would have assumed 10:1, tops. We get security reports, and sometimes they ask for a bounty, and only a very small number are bogus (but we don't have a formal bounty program). Less than half of our security issue reports are totally bogus, and another quarter are theoretical issues, but result in some sort of clean up in the code (e.g. no one can figure out how it could be exploited, but it gets refactored anyway).

    I've been meaning to try a formal bounty program, as our software is a high value target (administrative tool running on over a million systems), but we're Open Source and don't have a lot of budget for bounties or anything else. If it produced hundreds of reports for every valid issue, it'd be counter-productive, for sure.

    • tptacek  8 years ago

      The bounty prices won't be the problem. The constant negotiation over 100,000 different variants of unchecked redirection and login fixation will be the issue. Time is money.

      Hacker One should rename itself The Institute For Advanced Redirect Studies. I'm only partly kidding: bug bounty submitters are good at redirecting. Way better than I was before I started handling bounties. There's an interesting epistemological discussion to have about the low-value-yet-severity:critical bugs people file on bounty programs, because the level of cleverness required to exploit URL parsing differences between platforms is no less than what it takes to get an XSS bug.