Comment by jedberg
5 years ago
This is actually funny, because I was involved with the creation of this list, way back in 2004. The whole thing started as a way to stop phishing.
I was working at eBay/PayPal at the time, and we were finding a bunch of new phishing sites every day. We would keep a list and try to track down the owners of the (almost always hacked) sites and ask them to take it down. But sometimes it would take weeks or months for the site to get removed, so we looked for a better solution. We got together with the other big companies that were being phished (mostly banks) and formed a working group.
One of the things we did was approach the browser vendors and ask them if we could provide them a blacklist of phishing sites, which we already had, would they block those sites at the browser level.
For years, they said no, because they were worried about the liability of accidentally blocking something that wasn't a phishing site. So we all agreed to promise that no site would ever be put on the list without human verification and the lawyers did some lawyer magic to shift liability to the company that put a site on the list.
And thus, the built in blacklist was born. And it worked well for a while. We would find a site, put it on the list, and then all the browsers would block it.
But since then it seems that they have forgotten their fear of liability, as well as their promise that all sites on the list will be reviewed by a human. Now that the feature exists, they have found other uses for it.
And that is your slippery slope lesson for today! :)
This is an amazing story. It really demonstrates the way we pave our road to hell with good intentions...
We should really do something about this issue, where so few companies (arguably, a single one) hold so much power over the most fundamental technology of the era.
Here-here! I really wish there was more human involvement in a lot of these seemingly arbitrary AI-taken actions. Everything from app review to websites and more. This heavy reliance on automated systems has led us down this road. Shoot, keep it, just give us the option to guarantee human review - with of course transparency. We don't need anymore "some human looked at this and agreed, the decisions is final, goodbye."
I know it's easier said than done, especially when taking the scale of the requests into account, but the alternative has, does, and will continue to do serious harm to the many people and businesses caught in this wide, automated net.
It's interesting how closely the unfolding of this awful scenario has followed an entirely predictable path based on the shifting incentives: now hundreds of thousands of businesses face the same massive hazard of blocklisted without adequate human review, and with mediocre options to respond to it if it occurs.
Without a shift in incentives, its unlikely the outlook will improve. Unless the organisations affected (and those vulnerable) can organise and exert enough pressure for google to notice and adjust course, we're probably going to be stuck like this -or worse- for a long time.
3 replies →
It feels like the need for automated systems is a result of the ever-increasing size of the world (there are now nearly 5 billion internet users[0]). For Apple, app review can take days, mainly because doing human review [consistently] well and constantly for 8 hours a day isn't easy[1], leading to staffing issues when bad reviewers get weeded out and only a small percentage of hires stick around. Outside of hiring 10,000 employees just to endlessly review phishing links for 40 hours a week, you need automation to triage these phishing sites and deal with the outcome later such as via on-demand review by a human (which worked in this case, but won't always work - humans still make mistakes). I'm not sure if there is a solution for this problem outside of just not having the safe browsing product if 'makes no errors' is a requirement.
0: https://en.wikipedia.org/wiki/Global_Internet_usage
1: https://www.businessinsider.com/heres-why-it-really-sucks-to...
5 replies →
:s/Here-here/Hear hear/
1 reply →
> I really wish there was more human involvement in a lot of these seemingly arbitrary AI-taken actions.
Narrator: but it was only ever to get worse
Couldn’t agree more, the transparency is key. It enables faith in the system and outcome.
The counter argument to transparency will be that it provides too much information to those who aim to build phishing sites not blocked by the filter.
That said, we’ve experienced systems in which obfuscation wins out over transparency and it would be nice to tackle the challenges of transparency.
Are you implying that the list no longer has a good intention? I wouldn't be surprised if there are multiple orders of magnitude more phishing and hacked websites in 2021 than there was in 2004. Even with human checking, I doubt you'll even have 0% failure rate. Is the solution to just give up on blocking phishing sites?
The failure rate doesn't need to be 0%. If the solution is good, at least it'll be close to 0% which means that it'd be possible for the vendor to provide better support for the small number of mistakes so that they can be clearly explained to the affected party and rectified more quickly. If the failure rate is too high to make better support infeasible, then the current solution is not really a good one and we need to consider a revision.
> Are you implying that the list no longer has a good intention?
Most of the time I run into blocked sites they seem to be blocked because of copyright infringement, not phishing. The only phishing sites I've seen in the last year or so are custom tailored. For example, I had to deal with a compromised MS365 account last year where the bad actor spun up a custom phishing site using the logo, signature, etc. of the victim.
So IMHO the intentions are no longer pure plus the effect is diminished and being worked around.
The solution is for the legitimate sites that are driven out of business by Google AI to sue Google for tortuous interference and libel.
10 replies →
>Is the solution to just give up on blocking phishing sites?
IMHO yes. It's too much power for one company to wield. And especially a company with such questionable morals as Google. This cure is worse than the disease.
1 reply →
" Is the solution to just give up on blocking phishing sites?"
But maybe not do it by default on browser-level.
But if you do, then there really needs to be ways to combat wrong decisions in a timely manner.
The solution is simple: Liability. As soon as it becomes legally infeasible to let algorithms block people, it will stop happening.
Make it easy and affordable to submit legal complaints for tech misbehavior and make the penalties hurt.
Ah, so you suggest liability for the vendors of the software blocking websites, with, in practice [1], no liability for the operators of a compromised website, if it is phishing/malware?
This is a great approach, if your goal is to optimize for increasing the amount of dangerous crap on the web. But, eh, that's surely worth it, because the profitability of startups is more important then little things like the security of the average netizen...
[1] Even if you make the operators liable [2], in practice, you'll never be able to collect from most of them. Whereas the blacklist curators are a singular, convenient target...
[2] If you can demonstrate how the operators of compromised websites can be held liable for all the harm they cause, I will happily agree that we should do away with blacklists. Unfortunately, the technical and legislative solutions for this are much worse than the disease you are trying to treat.
4 replies →
This was the case with railroads too, only a few controlled the biggest and most transforming and business-integral tech of 1800s.
Prior to that it was those that controlled the printing presses.
...
History continues to repeat itself.
2 millions phishing sites and counting... with 40000 websites added each week.
https://transparencyreport.google.com/safe-browsing/overview...
I guess the automation started in 2007 or so.
Like some kind of perverse blockchain, no site is ever removed, even though most phishing sites don't live long.
I think you mean 2017? 2007 is when the feature launched.
january/february 2007 looks like the time the list jumped from a few hundred to tens of thousands of sites.
2 replies →
Something similar I've just read in zero to one (by Blake Masters and Peter Thiel). Peter argues that computers can't replace humans - it'd be foolish to expect that at least for coming decades – strong AI replacing human is the problem of 22nd century. He proposes Complementarity and provides a successful implementation of this idea in PayPal fraud detection system way back in 2002 when purely automated detection algorithms were quickly overcome by determined fraudsters. He went on founding Palantir based on the same idea.
>>> In mid-2000, we had survived the dot-com crash and we were growing fast, but we faced one huge problem: we were losing upwards of $10 million to credit card fraud every month. Since we were processing hundreds or even thousands of transactions per minute, we couldn’t possibly review each one—no human quality control team could work that fast. So we did what any group of engineers would do: we tried to automate a solution. First, Max Levchin assembled an elite team of mathematicians to study the fraudulent transfers in detail. Then we took what we learned and wrote software to automatically identify and cancel bogus transactions in real time. But it quickly became clear that this approach wouldn’t work either: after an hour or two, the thieves would catch on and change their tactics. We were dealing with an adaptive enemy, and our software couldn’t adapt in response. The fraudsters’ adaptive evasions fooled our automatic detection algorithms, but we found that they didn’t fool our human analysts as easily. So Max and his engineers rewrote the software to take a hybrid approach: the computer would flag the most suspicious transactions on a well-designed user interface, and human operators would make the final judgment as to their legitimacy. Thanks to this hybrid system—we named it “Igor,” after the Russian fraudster who bragged that we’d never be able to stop him—we turned our first quarterly profit in the first quarter of 2002 (as opposed to a quarterly loss of $29.3 million one year before). The FBI asked us if we’d let them use Igor to help detect financial crime. And Max was able to boast, grandiosely but truthfully, that he was “the Sherlock Holmes of the Internet Underground.” This kind of man-machine symbiosis enabled PayPal to stay in business, which in turn enabled hundreds of thousands of small businesses to accept the payments they needed to thrive on the internet. None of it would have been possible without the man-machine solution—even though most people would never see it or even hear about it.
Liability was my first though. How is an assertion that a site contains malware not libel? Site would be easily able to demonstrate lost revenue.
Can someone dig out that old agreement to see if Google can be sued big time for this?
I doubt it but I must say it would make me happy and that would be weird because Schadenfreude normally isn't my thing.
> since then it seems that they have forgotten their fear of liability
They most likely have offloaded the liability to a “machine learning algorithm”. It’s easy for companies to point the finger at an algorithm instead of them taking responsibility.
Which then leads them to the awkward place of having to be transparent about how their algorithm work
Either take responsibility, or be transparent.
But we all want our cake and eat it
I take offense to this. Sure, I like to eat cake.
But if I liked to eat cake as much as Google does, I'd have died of obesity (= have my life ruined by legal issues) a long time ago.
Simple solution = let google use their, imperfect (false-positives) filter, allow them to collect $12 / year not to be blacklisted, and google to send all revenue to the Electronic Frontier Foundation or similar internet defending foundations.
Another road to hell paved with good intentions. Once everyone’s paying, who’s to stop them from pocketing the money instead?
“After careful review, we’ve concluded that the Electronic Frontier Foundation no longer aligns with the goals of Google or its parent company Alphabet Inc. to the extent we require from recipients of our Freedom Fund. We will place these funds in a separate account and use them in ways we believe will be in the best interest of digital freedom, both now and in the future.”
Worse, getting such money flow, EFF will get corrupt very soon.
1 reply →
"For years, they said no, because they were worried about the liability of accidentally blocking something that wasn't a phishing site."
Can anyone explain how a web browser author could be liable for using a blacklist. Once past the disclaimer in uppercase that precedes every software install, past the Public Suffix (White)List that browsers include, how do you successfully sue the author of a software program, a web browser, for having a dommainname blacklist. Spamhaus was once ordered to pay $11 million for blacklisting some spammers, but that did not involve a contractual relationship, e.g., a software license, between the spammers and Spamhaus.
I think the situation is actually exactly like the Spamhaus case you describe: it wouldn't be the browser user that sues, but the blocked website's owner. The website's owner need not have accepted any kind of agreement from the browser maker in order to be harmed by the blocklist.
Perhaps the website would sue the author of the list.
That does not explain why this comment suggests a browser author was afraid to use the list.
The browser author could easily require the list author to agree that the browser author has no obligations to the list author if the list author gets sued by a website, and the list author must idemnify the browser author if the browser author is named in any suit over the list. The list author must assume all the risk.
That's very interesting. Would you not think for a moment that such mechanism could be abused?
The internet was a much kinder trusting place back then. We assumed when the browser makers agreed to not use it for bad things, we believed them.
I think as always great ideas do not account for human nature...
1 reply →
It’s amazing that a person can write this and not feel any remorse for the pain they caused.