Comment by grugq
7 years ago
It is not illegal to sell that type of software. It is not a black market, it is a grey market.
There is no way you will ever hear authentic answers to your questions. The only time anyone tried to explain that the resulting article backfired on the interviewee. (Disclaimer, it was me)
Governments do not buy from developers. The paperwork would be insane. They buy from businesses like Raytheon. How Raytheon gets them is opaque. But they do employ hundreds of exploit developers. Read the r/netsec job postings and notice how many require having a TS clearance. Every interesting job that says “work on vulnerability discovery and exploit development” requires TS.
Governments generally speaking do not cheat on business deals where they want to continue having access to that market. It is like stiffing the company that sells you replacement parts for your government vehicles. You save money now, but in the future your planes can’t fly and no one will do business with you.
All of this I explained during the interview, but the objective of the article was not what I assumed it would be, which was to address the dynamics of how the market works. I was naive to think that, but in my defense I was genuinely shocked that people were unaware of the market (it has existed forever). Literally everyone who is an infosec rockstar has been involved with exploit sales [0]. Many still are because It allows them to work on what they enjoy — bugs and exploits — and remunerates for their expertise. They get paid a living wage to do what they want. Like any freelance developer. They are just smart enough to keep their mouths shut.
I haven’t been involved with the market for almost a decade now but you’ll still hear people saying shit like “how does it feel to sell weapons to dictators??” (Even On here there are a number of such comments.) I can truthfully answer that I have no idea. I only ever sold software to western governments who had a hard on for terrorists.
I’m still angry about it, but I have no one but myself to blame. You can’t unfuck the goat. C’est la vie. People want sensational stories about evil people, they don’t want stories about the dynamics of a grey market software industry. No one will ever speak about it again (lessons learned analysis! Protip, don’t be the lesson others learn from).
The market has changed massively over the years. It is nothing like the one I was involved in back then. However, as I said, no one will ever discuss it again. They saw what happened and they won’t speak in public about it.
What was, is, and will continue to be, the legitimate sale of vulnerabilities is now closed forever.
As a thought experiment, think of this. Let’s take it for granted that the IC counter terrorist units and the legal authorities hunting for child abusers are acting in good faith. That is, not every single person at NSA is desperate to see what you are doing on the Internet (literally, you are noise obscuring their signal). There are people who are going after child sex abusers, do you want them to have the capability to exploit a web browser or do you want web browsers to be safe tools for child abusers. This is not hypothetical [1].
There cannot be a discussion about a market where there is so much hysteria about fringe cases of abuse. Rather than trying to find ways of mitigating against abuse, the reaction has been to advocate for prohibition. Prohibition does not work, it simply drives reputable operators out of the market.
The conversation about vulnerability sales has been as even handed and rational as the conversation about marijuana in the 50s. Instead of marijuana madness you get “the FBI can hack your computer!!” ...I guess the upside is that at least this time the topic is not a proxy for racism [edit: I retract that statement. Pretty much every rationalization about banning vulnerability sales talks about African or Arabian buyers.]
And again, I have said too much. Try to explain something, get called a baby killer. I’ll bet there will be accusations of enabling dictators to spy on civil rights activists. To preempt the “you don’t know what happens after you sell it!” I say simply this — the point of having a middleman to handle the transaction is to ensure that you sell to the right end users. Exploit developers don’t want to sell to dictators, they find someone who can get them access to a market where their work will be used ethically. That can’t be said for all, of course. The jailbreak community in particular is essentially a vendor to the Chinese government.
But there you go. The most you’ll hear about it from someone that actually knows what they’re talking about.
[edit: haha, see? It was brought up before I even posted a response! [2] There is no accurate information. Literally every single paper on the topic cites newspaper articles rather than academic research. This is actually unique. It is the outlier case. Mara did a review of the literature and found that the majority of citations were to articles, far in excess of other topics)
[0] https://news.ycombinator.com/item?id=20651348 .. feel free to read the article and think what you like. Andy Greenberg is a good journalist. I was an idiot. ¯\_(ツ)_/¯
> the resulting article was a hatchet job
Was that the forbes article linked above?
> You can’t unfuck the goat. C’est la vie.
That goat laid you golden eggs though. It takes me over 15 years to earn a $1m paycheck, and I wouldn't mind dealing with some people moaning at me for it. People always find something to complain about anyway, so I wouldn't be too concerned about it.
> The conversation about vulnerability sales has been as even handed and rational as the conversation about marijuana in the 50s.
Sure, you're right, but this is true for many new things, you're just smack in the middle of this discussion. It's good to have these discussions though, because it makes people aware that otherwise weren't. Comparable hysteria is currently happening with 'company X is listening to your conversations' and 'self driving cars might kill you to save a baby'. People in the field have been discussing the ethics of these things for a long time, but now it's becoming a public discussion. This happens when things grow.
Personally, I prefer the black/grey zero day market over back doors being proposed by some. Those would be permanent vulnerabilities, while zero days are (usually) temporary and probably only used while absolutely necessary, opposed to just eavesdropping on anybody. I also see the need, because the internet gives bad people too many places to hide.
So, from me, thanks for your services, you probably helped keep us safe from bad actors.
Yeah, that’s part of the hatchet job. I said I was projecting sales of $1M over the year. At 15% commission that would be $150k. You can make a lot more money than that, I’m sure. Also, don’t predict your sales funnel in February when you have no historical data to compare it with. I was off by about $900k.
So yeah, that $15k golden egg. ¯\_(ツ)_/¯
At thetime I did not know about phrases like “off the record” or that you could have corrections made to articles that had false information. Had I known I would have OTRed at the beginning although I should never have spoken in the first place. And I should have made them correct the inaccuracies.
But, c’est la vie.
> I was projecting sales of $1M over the year. At 15% commission that would be $150k.
That's a pretty big difference indeed, that's more like a normal SV salary than a 'live for ever in Thailand' amount of money.
2 replies →
Damn this is an awesome break down of the industry, and it's hilarious to me that lo and behold someone suggests the Greenberg article and yeh does grugq himself turn up to settle the score.
I can't think my way around your point about prohibition though - I think someone saying "selling exploits is bad" is also someone that would say "the government shouldn't be monitoring us, pedophile or not," and that's part of why they don't think exploits should be sold to governments. Could be wrong.
But, we all generally seem to feel that the government shouldn't be given back doors into our devices that only they get to use, yeah? So instead the alternative is an endless arms race as chrome or whoever tries to out engineer the FBI? Why not just give them the backdoor at that point? (I.e., why not just support them having the backdoor, I'm not implying those in your wheelhouse have the power to legislate or anything)
Think of it like GMO. There are two sides with legitimate concerns. But only one side can speak publicly.
As for backdoored Chrome, what is to prevent China using a modified version of Firefox that removes the backdoor? It would blind NSA to collection on the Chinese target.
There is no way you can use backdoors against hard targets. Hard targets are why they need 0day. It is an arms race because it is a conflict between states.
Whatever fears people have about 0day being used against them are, as I’ve said before, like worrying about ninjas rather than cardio vascular disease. One is something you have no control over, but almost no exposure to as a risk. The other requires regular work to stay safe.
Years ago I wrote “free security advice” and the basic concept is still relevant. I should update it now though. Android 9 is a much harder target that 4.4 was. I would actually rate Android as safer than iOS because all of these ridiculous articles about million dollar pay outs have driven most developers towards iOS, and iOS is a monoculture.
A hardened Android device (disclaimer, I’m making one for retail sale) is safer than a stock iOS.
Literally everything in the media is complete garbage. No one who knows how things work would ever discuss them again.
Your argument is limited to technical and political science concepts, and by limiting itself so, is correct. It is inapplicable to the real world.
Governments have used zero days. Most famously to use a zero day unlock an iPhone against a terrorist (whose house was ransacked by the news media). Less famously was to botch a legal case against a pedophile (amazingly, it would be possible to find and arrest nearly all pedophiles on Tor by burning half a million dollars in zero days). But the government didn’t want to release the zero day for Play Pen and Mozilla got involved in the case.
But Freedom Hosting’s zero day was discovered while it was being used. I think the government still uses zero days, but parallel constructs the evidence from them. This is policy making by mismanagement.
On face value, the government is involved in abhorrently irrational decision making. The government cannot be considered responsible enough to have zero days, but that’s an argument that will lead nowhere.
5 replies →
>Years ago I wrote “free security advice” and the basic concept is still relevant. I should update it now though.
It looks like you DID update it: https://gist.github.com/grugq/353b6fc9b094d5700c70
And from that gist:
> Use an iPod or an iPad without a SIM card
> Use an iPhone
How can you then says:
> A hardened Android device (disclaimer, I’m making one for retail sale) is safer than a stock iOS.
?
Is Android (and in general any open source system) safer than a iOS (a closed and highly customized system) ?
The idea I heard over and over is that a open source system is more secure because the code is scrutinized by anyone that wants to.
But with a monthly security update and how quickly a vulnerability can be exploited, it does not seems to be the case anymore.
The main reason is the time between a vulnerability is patched in the source code and the patch is deployed. When a commit that fixes a vulnerability is committed on the Android codebase, anybody that knows what is looking at would be able to notice it, and likely build/distribute an exploit before the patch is actually pushed to all users. On a closed source system, an exploiter can still reverse engineer the changes in an update but less people have the skills to do it and it is not straight forward to understand which changes in the code are a security patches.
Considering the timing and what I see on the Android security bulletin almost every month there are EoP and even RCE vulnerabilities being patched. A Google Phone, on average, will go 2 weeks every month vulnerable to a "known" vulnerability.
For all the others the situation is dramatically worse. Samsung is at best a month behind the security update schedule. A Samsung's user will have a phone that is always behind the last vulnerabilities patched and visible in the Android code base.
Some of these vulnerabilities can be quickly distributed since everybody has an LTE internet connection, read new on a browser.
1 reply →
> A hardened Android device (disclaimer, I’m making one for retail sale)
Any more information on this? I'm more than a little depressed by the current options in phones - I don't relish the idea of moving to ios - but at the same time I'm a bit worried about the direction Android is taking...
What kind of marked/price range are you aiming for?
> What was, is, and will continue to be, the legitimate sale of vulnerabilities is now closed forever.
So in past, present and future, the legitimate sale of vulnerabilities is now closed forever. When was legitimate?
Are you saying that since it is not legit, exploits should never be sold? What are you advocating for ?
I think (though I can't be sure), what they're trying to say is that it's still legitimate, but it's opaque, because nobody wants to talk about it. Because of that, it seems, to outsiders, like it's an evil black market, even though many people involved in it, believe that they're doing the right thing.
Exactly so.
Sorry, to clarify I was referring to the voice of industry insiders. I mean that no one who knows is willing to speak about it.
There is so much bullshit about the “highly lucrative black market” it is staggering. The market is not big. There is significant risk which gets factored into the payment structure, so the payments are lower than people imagine.
The market is not very liquid. If you have a Chrome capability for sale but your client already has a Chrome capability, they won’t buy it. If their capability dies, then they’ll want yours, but by then yours might be dead as well. Gross oversimplification, but that is generally how things work. The demand is very specific, the supply is very limited, and the product is very fragile (particularly time sensitive.) It is lucrative like making a startup is lucrative. You invest a lot of time and resources and sometimes, with luck, you win big, but the odds are not in your favor for a million dollar payday.
Most articles treat it like some sort of open market drugs bazaar. It is nothing like that at all. It is more like a handcrafted goods faire with a few wealthy customers looking for exactly the thing they need. Only they won’t tell you what they need, they simply want to see what is on display. Lots of window shoppers, as it were.
The product has an unknown shelf life.
The customer cannot tell you what they need, they will only look at what you have and possibly choose something.
For the developer they need to ensure that they provide sufficient information about the capability so the customer can make an informed decision. But they have to avoid revealing sufficient details that it can be reproduced from the ad copy.
Part of what a broker does is actually translating between two parties who don’t speak the same language. The customer needs a tor browser Bundle capability. The developer has written a UAF RCE Firefox that relying on JIT spraying for reliability. Someone has to translate from exploit dev speak into IC language.
For the IC, that TBB capability is a replaceable part in a larger program that enables them to achieve their mission objectives. For the exploit dev, that bug is a labor of love that they spent months working on. They have completely different views on the value of the capability. One side sees it as a component they need for a machine they want to use. The other side sees it as weeks of frustration and pain invested into a unique masterpiece.
They have different expectations, don’t speak the same language, and don’t trust each other. Things have changed a lot from when I was involved. It’s all very fascinating but, as I said, no one who knows about it will discuss it.
I’m being stupid and talking about it, again. But hopefully this will clear up some of the stupid myths about the vulnerability market.
For example all those “wow, a way to read a someone’s private messages on Facebook? That’s got to be worth millions!!” No, it is not. If a legitimate client wants to read someone’s messages on Facebook, they get a warrant. There is no ROI for cyber criminals, and whatever it might be worth to North Korea the risks associated with that sale are not worth it. That bug is worth whatever Facebook says it is worth. Dropping the 0day would make for some news, but mostly it would be negative. So the only rational way for a security researcher to make money from a Facebook bug is through the bug bounty system. (I’m not addressing cyber criminals discovering such a bug, because that is not relevant to the issue of vulnerability sales.)
Based on their recent acquisition, it seems like Azimuth made something of a working de-risked business model relative to the uncertainty of the broker days, no?
I appreciate your honesty and taking responsibility. The time I spent in security research had me putting blame pretty far away from middle-people: (a) the users and buyers who almost exclusively go with insecure crap, even if secure ones are highly-usable and/or free; (b) the developers who do nothing to make their software secure. On (b), some vulnerabilities could've been prevented with push-button tools like AFL that they just don't bother to run. Fish in a Barrel LLC makes that point more comically. Those groups driving the vulnerabilities would have to get their shit together before folks selling them become truly bad to me. Do have two points to address, though.
"Literally every single paper on the topic cites newspaper articles rather than academic research."
You mention that everyone is doing it with no citations of academic sources. I'd be interested in reading any recent research you believe is high quality and represents the current market. That other paper was dated 2007. I figure there's been some changes.
"Let’s take it for granted that the IC counter terrorist units and the legal authorities hunting for child abusers are acting in good faith. "
We can't take that for granted. Ok, so the prior precedent I pushed Schneier et al to use in media was J Edgar Hoover. He used blackmail on initially a small number of politicians in control of his budget and power to massively increase his budget and power. The Feds committed all kinds of civil rights abuses. His reign lasted a long time with his power growing. He accomplished it all through surveillance using ancient methods that required actual people listening in on calls and such. Both Feds and I.C. stay doing power grabs even though some or all of that was stopped. We'll never know since FBI continued to have its budget and power.
I predicted that, post-9/11, they'd do a power grab as a USAP. If it's a USAP, then only a few in Congress can oversee the program and therefore only a few need to be controlled. Sure enough, Snowden leaks confirmed they did that for nation-wide surveillance, Congress kept doing nothing more than they usually do (didn't even read reports per GAO), gave them retroactive immunity for abuses, the warrants weren't for specific individuals ("targeting criteria"), they shared data with all kinds of non-terrorism-related agencies, at least one (DEA) regularly arrested folks after lying about sources, and they're steadily expanding that. Again, with criminal immunity for whatever secret things they're doing.
So, no we can't consider them acting in good faith. They've constantly lied to Americans and Congress about programs that are used to put people in jail for all sorts of stuff. There's no telling what they'll do if we give them too much power. That's why some of us advocated warrants for information or specific acts of surveillance. One can also hold people in contempt for not giving up keys. It needs to be targeted with evidence behind what they're doing.
And, yes, some horrible people will get away with crimes like they do with our other civil rights. You'd have to be non-stop spying on every person anywhere near a child 24/7 to achieve the goal of preventing that. Yet, we don't do that because we as a society made a trade-off. This is another one. This isn't hypothetical: the FBI is so corrupt they pay people to recruit/bust terrorists with Presidents and Congress usually taking bribes from companies to get elected. We should always treat them as a threat that acts in their own self-interest that might differ greatly from ours.
> I appreciate your honesty and taking responsibility.
It is a fatal character flaw I have. When people want to know about something I try to help them.
> You mention that everyone is doing it with no citations of academic sources. I'd be interested in reading any recent research you believe is high quality
There was one by RAND which is good.
https://www.rand.org/content/dam/rand/pubs/research_reports/...
> represents the current market.
There is nothing that I am aware of that discusses the current market. The RAND paper is closest.
> acting in good faith.
I should not have used a blanket statement. My point is that there are people in IC who are legitimately going after terrorists and child abusers. They have a legitimate need for capabilities that enable them to do that.
I am not saying that the IC is a benign and wonderful government organ. I am saying that within IC there are people who are actually hunting terrorists and pedophiles. I didn't want to explain all of that because it is obvious that it is true. Hence, "lets take it for granted". Rather than discussing the history of the IC, I wanted to explain that there are legitimate uses for 0day and that is the issue being discussed.
The rest is not relevant to explaining how the vulnerability market operates. (Well, how it did in 2011.) When someone asks "how do shares work?" you don't start off by talking about boom and bust markets and macroeconomics. Same thing here. "How does the market work?" is not a question about the IC. It is about how the market works. If you're talking about the vulnerability market you talk about the vulnerability market. You have to assume that there are legitimate players who are acting in good faith.
This entire post is why I abridged it to "lets assume good faith."
Leaving it out because it was mostly irrelevant makes sense. There's definitely folks doing good with these capabilities. I'm a big fan of their work and grateful for their sacrifices. And thanks for the RAND link. I'll check it out later.