Comment by brians
8 years ago
Nah. Bug bounties don't work for services like CDNs. Maybe they do elsewhere. But for enterprise services, the noise rate is too high, and the very good bug finders are either salaried, free, or working for the adversary.
I think I'd need to see some sort of evidence of this assertion. Bug bounties are commonly offered across a huge variety of online services, and they get results...not always, not necessarily consistently high quality, but even the giants (facebook comes to mind) have had reasonably serious bugs found by people seeking bounties.
He's not wrong about the noise level. I conducted a survey of the most notable bug bounties in 2014 and found that the largest companies either have ineffective programs or quickly scale teams to handle inbound reports full-time. There are security engineers at Google and Facebook who spend a majority of their time responding to, and triaging bug bounty submissions.
That said, I disagree that bug bounties don't work for CDNs. You can scale a bug bounty up, it just requires resources. Cloudflare has those resources, and part of it is a function of the reward tiers you offer.
Bounty researchers aren't the only quasi-rational economic actors in this sytem. Cloudflare, we might surmise, get enough benefit from their bounty program that they're willing to pay for its administration costs and the occasional T-shirt, but they don't see value in spending more than that.
More than that, access to the service is actually the limiting factor for good bug bounty results. Cloudflare's bug bounty, we might surmise, works as well as it does because anyone can sign up for a Cloudflare account for free. For an enterprise CDN, who won't talk to a potential customer without the prospect of an $x0,000+/year contract, everyone who has enough access to the service to, in the general course of business, find and submit meaningful reports is employed by a customer, and likely prohibited from accepting substantial rewards. Everyone else either doesn't have enough access to submit meaningful reports, or the bug is so bad (like this one) that they'll report it regardless.
Arguably this shows that Cloudflare and other CDNs are right in their calculations: Tavis disclosed this bug to Cloudflare without promise of a payout, or even a T-shirt. Might some good Samaritan on the Internet have noticed the bug and reported it earlier if the bounty was more substantial? Perhaps. But in responding to a vulnerability of this magnitude, you want to work with someone of Tavis's caliber, who has the good of all the stakeholders in mind, not a profit-motivated rando.
I'll gladly offer some anecdotal evidence:
We've got about 2500 tickets in our ticketing queue that have been filed over the past 8 months (excluding spam). Out of those 2500 tickets, only five are valid issues, and only one came with an actual write up.
The signal to noise ratio is absolutely awful - and it's not uncommon for people with invalid issues to demand that you pay them regardless.
Wow, that's much worse than I would have guessed. I would have assumed 10:1, tops. We get security reports, and sometimes they ask for a bounty, and only a very small number are bogus (but we don't have a formal bounty program). Less than half of our security issue reports are totally bogus, and another quarter are theoretical issues, but result in some sort of clean up in the code (e.g. no one can figure out how it could be exploited, but it gets refactored anyway).
I've been meaning to try a formal bounty program, as our software is a high value target (administrative tool running on over a million systems), but we're Open Source and don't have a lot of budget for bounties or anything else. If it produced hundreds of reports for every valid issue, it'd be counter-productive, for sure.
2 replies →
> Nah. Bug bounties don't work for services like CDNs. Maybe they do elsewhere. But for enterprise services, the noise rate is too high, and the very good bug finders are either salaried, free, or working for the adversary.
Yes, running a real bug bounty system requires professional security engineers and a professional security posture to sort through the noise. However, when the sole product you are selling is security (i.e. Cloudflare) you kind of have to admit it should be expected that they do so.
It isn't "too high", it simply requires a serious financial commitment to security in the terms of salaried security engineers.
As to your other point, No one works for free. Project Zero is paid for by Google. Security engineers are going to prioritize the purposes that make them real, hard cash.
Here's a question: what's the trade-off in terms of return on investment between hiring salaried security engineers to administer a bug bounty and hiring salaried security engineers to find bugs directly?
Parent's claim, as I read it, is that it's a better use of an enterprise CDN's money to hire security engineers to find bugs than to administer a bounty. Seems plausible to me. Where's that line?
> Parent's claim, as I read it, is that it's a better use of an enterprise CDN's money to hire security engineers to find bugs than to administer a bounty. Seems plausible to me. Where's that line?
Depends on the company, but tbpfh, most security engineers in a group tend to have a culture and that culture creates common blindspots. The fact they weren't testing for this sort of issue (i.e. parser memory leaks) is an example of something that seems obvious to some people that others ignore.
Maybe that is just my experience tho.
Facebook and Google have bug bounties. That's pretty big scale.
Facebook and Google are not, at base, enterprise services.