Comment by AceJohnny2
5 years ago
If you've got an automated vetting process with a 99.999% success rate, but are dealing with billions of accounts, that's still tens of thousands of false positives.
At that level, "percentage" is an insufficient measure. You want "permillionage", or maybe more colloquially "DPM" for "Defects Per Million" or even "DPB".
You'll still get false positives though, so you provide an appeal process. But what's to prevent the bad actors from abusing the appeal process while leaving your more clueless legitimate users lost in the dust?
(As the joke goes: "There is considerable overlap between the intelligence of the smartest bears and the dumbest tourists" [1])
Can you build any vetting process, and associated appeal process, that successfully keeps all the bad actors out, and doesn't exclude your good users? What about those on the edge? Or those that switch? Or those who are busy, or wary?
There's a lot of money riding on that.
[1] https://www.schneier.com/blog/archives/2006/08/security_is_a...
I think this is a balancing act of risks, and I wanted to bring up what I believe to be a success story when it comes to handling suspensions: Microsoft.
One thing I believe Microsoft gets right is that suspensions are isolated to the service whose TOS was violated. I.e. violating the hotmail TOS doesn't suspend you from their other services. I think this makes the impact of a false positive less catastrophic, while still removing actual problematic users from the service. This may be an artifact of how teams work together at Microsoft.
Yup, I agree this is the better solution. The monolithic "one account rules everything" approach just increases the user's vulnerability.
It's largely what made Facebook's forcing usage of their account for Oculus users so ass-backwards.
> This may be an artifact of how teams work together at Microsoft.
It may be an artifact of Microsoft actually being regulated for monopolistic practices.
There's nothing at all in the old DOJ settlement that imposes anything like this.
1 reply →
If we did that at Microsoft when we were bringing Hotmail under the MS umbrella, DOJ would have ripped the company into 10 pieces
If you're implying that there's just no way to support their users then I'm going to disagree.
At Google's scale and profitability, saying you can't build an appeals process that supports your paying users is just ridiculous. And at this point the collateral damage to Stadia's already tenuous reputation is going to be a lot more than paying someone to vet him manually.
Honestly, the answer is to charge people a fee, in order to appeal a ban. A fee that covers the cost of investigating the incident, making it revenue-neutral. This way, Google would have every incentive to investigate thoroughly all appeals, including repeated appeals by the same person.
From the user's perspective, it's still a pretty good deal. There's a 99.999% chance that you get to use gmail/youtube/etc for free. And a 0.001% chance that you'll end up a statistic, and need to pay a nominal fee for an appeal.
Unfortunately, I don't think the above will ever happen, because it would be a PR nightmare. "Google wants to charge you money, just to appeal a ban!" It's still better than the status quo, where people have almost no recourse when they are banned. But it still sounds way better in the media, if you just pretend as though these things never happen. Hence the status quo - use automated systems to cheaply get to a 99.999% success rate, and spend as little money as possible on the remaining 0.001%
So now banning people incorrectly is a revenue generator?
The answer is to force google to be open and more transparent through regulations and have to scale up to deal with it and eat into their profits.
The assumption up front should not be that we need to care about protecting their profits.
Absurd but not new. Equifax was charging people to freeze their credit for a while after the breach, until public pressure mounted.
They probably made a TON of money off of that, and off the credit protection services they offer directly or through subsidiaries.
> So now banning people incorrectly is a revenue generator?
It need not be, as long as the fee is less than the cost. It could be symbolic (say $1). But the problem is that it would be seen as a revenue generator whether it is or not.
They don't even have to keep the fee of the query is legitimate. They can reimburse it or keep it in the user's wallet when they consider that this was either a false positive or a honest mistake. The cost would be minimal but would deter a lot of people trying to game the system.
I completely agree.
And if companies don't want to do it, that should be easy to regulate though. Requiring a human centric appeal process even if it has a fee, and prohibiting blanket account bans (if you get banned on gmail it doesn't affect your android and play store accounts, for example)
There are other provisions that I consider important like not being able to reuse email addresses and requiring the forwarding of email for at least 6 months after any account termination (getting banned from your email address can have disastrous consequences)
Google One, which costs 2$ a month, as far as I can tell provides you with some level of support, which is definitely more than zero.
Do you even have access to your Google One support if your Google account is banned?
The problem with unjustified bans due to some algorithm is also: These cases might not even be a close calls like: “oh yeah this person did something that is in the grey area of what our policies state. I will ban him but he might interpret things differently.”
No if you enforce your policies strictly by (machine learning) algorithms it could just be a matter of misinterpreting a different language, slang, irony or something else. Which makes these bans even more infuriating.
The lesson here is: you are too big. If you were smaller, you could manage these issues. But you choose to be big instead.
Counter-example - Amazon. You can reach someone at Amazon and they are ginormous too.
Counter-counter example, even if you do reach someone at Amazon they're not necessarily going to do anything useful.
I've had a problem with my Amazon account for years now, after Amazon billed me (on my seller account) for something they shouldn't have.
After I complained, they agreed to refund it. Except the refund never arrived.
Asked many times over the years "WTF?", and someone always promises to look into it after agreeing they can see the problem.
Never to be heard from again. Same pattern has happened every single time (many times). Obviously, something about it puts it in the "too hard" basket... :/
Needless to say, I don't use Amazon's services much at all any more unless required for job purposes. And I steer people away from AWS for the same reason too.
2 replies →
Is this really true? If Gmail was replaced with a dozen competing services each with "only" 100M users each, would the total number of moderators be lower? How does the number of required human moderators per million users scale, and why?
I agree: not true. The advantage of automation is you can do more for less which extends the reach in wealth and services available to the human race. Automation is a beautiful thing and gmail being too big to service with human support is not understanding that we'll never have enough intelligence power to police every square inch of existence + the net if we rely solely on human intelligence.
Problem is: can we cultivate machine learning intelligence to be as good as some of the best human arbiters?
https://www.youtube.com/watch?v=91TRVubKcEM
3 replies →
But you'll have the option of switching to one of those other companies.
You can't choose to stay small unless you're someone like clubhouse which still has a long waitlist for sign-ups, and even then they're trying to build their infrastructure wide enough to accompany everyone. Not offering service to all/99.9% of potential customers is effectively lost value and goes against shareholders' expectations.
That's like saying a restaurant can't choose not to serve a billion people even though it only has enough capacity to seat and make food for 20: if you can't provide legitimate service for everyone, you need to not allow more people. The core problem here is that users keep signing up for Google services without being informed correctly ahead of time why that's idiotic, and the only fix for this is going to be regulatory: either Google needs to change how they handle banning people (there should be some law that if they accepted responsibility to store someone else's data that they have some minimum retention time for it letting you access it or something), come up with a working appeals process (and ensure that they have enough employees to handle the expected appeal load before either signing up new accounts or banning old ones), or they need to be forced to have a giant sticker on the box with a skull and crossbones on it which says that the moral equivalent of the surgeon general needs you to be informed of the serious risks that are associated with using this ridiculous service offering.
Then lets regulate size if the market is going to push companies towards inhumane choices.
>If you've got an automated vetting process with a 99.999% success rate, but are dealing with billions of accounts, that's still tens of thousands of false positives.
Doesn't matter. If you're dealing with billions of accounts then you're earning billions of dollars. Just hire more people. Scale must never be an excuse for poor customer service.
It depends on the unit economics.
Google has billions of accounts because it is FREE create them. Which could mean the cost of providing human support is actually too expensive on a per unit basis. The only way to rectify these economics is to charge for the account.
I pay for Google One to store more photos...however I have no clue if this improves my situation. Does the algorithm give me more slack for being a long, paid user? Do I get real customer support in the event I do get flagged? No clue.
> You can't even trust phone companies to do their job right and ensure the secure verification code is sent to the right phone! You provided some more secure ways for users to authenticate themselves,
For those that don't know, phone companies are easily susceptible to sim-swapping attacks which can make it easy for an attacker to intercept SMS 2fa: https://news.ycombinator.com/item?id=22016212
Edit: looks like OP changed their entire comment while I was replying.
You can totally trust phone companies to "do their job right". You need to understand what their job is though.
The Telcos never signed up to being a "secure verification code provider". Almost a decade ago, the local Telco industry group told us all:
"SMS is not designed to be a secure communications channel and should not be used by banks for electronic funds transfer authentication,"
https://www.itnews.com.au/news/telcos-declare-sms-unsafe-for...
Any company that uses SMS for 2FA is offloading risk and security to an industry that never expected it, and explicitly seeks to not provide it.
A Telco _desperately_ wants to be able to get you back up and running (making calls and spending money) on a new phone using your existing number before you walk out of the shop. And even more, they want to be able to transfer you across as a customer from a competitor - and have your existing number work on their network.
"Sim Swapping" is a valuable feature for Telcos. They have significant negative incentives to make it difficult. They don't want to secure your PayPal account, and nobody (least of all PayPal) should expect them to do a good job of it, certainly not for free...
Yeah sorry, I thought the original version was overly flowery, and the same point could be made more succintly.
> Can you build any vetting process,
Yes, it's pretty simple. Create and enforce some consumer protection laws which require, for example, that any company larger than a certain size is required to establish support offices staffed by humans in every major town. And required to resolve every issue within X days either by fixing the problem or clearly documenting why not. If not, no arbitration allowed, so they are subject to lawsuits if the reason doesn't hold scrutiny.
Problem solved. Companies like goog, facebook et.al. can easily afford this and it'll stop this ridiculous behavior.
It also to some extent protects the companies. Spambots who create a million accounts can't replicate a million humans to show up at the support office, so it establishes a human:human relationship that's completely missing today.
This would all be perfectly okay and understandable if the AI were the first line of defense and there was any meaningful way at all to contact support and escalate things after that filter. (I mean besides making headlines in all the gaming-news articles.)
> But 0.001% of billions or users is still millions of accounts...
Not that I disagree with your point, but even if we assume 50 billion accounts (6+ for every human on earth), 0.001% of that would still be 'just' 100k, not millions.
Oops, quite right. I multiplied by 0.001 when it should've been 0.00001 (because percent) >_<
Fixed
Yes there is a lot of money riding on that, but that is the cost of doing business.
Why banks have heavy compliance costs? Doing proper AML and KYC costs money and society decided that it was critical enough to bear that cost even in light regulation countries.
A lot of the financial success of those companies is in part the result of not fully taking responsibility for the consequences of their business activity. Eventually they will, under social pressure that this post success represent, or by laws.
At some point percentage is insufficient, but it's because it's a rate. Permillionage/DPM doesn't fix it. It's the number of people affected that matters, so if you have it at 99.9% and grow 10x, you ought to improve it to 99.99% to not become eviler. If you just stay at 99.9% when you grow 10x, you're harming 10x the people.
I'd use the total number of false positives as the proper measure.
If a company has so many users that it can't hire enough employees to manually handle the false positives properly, it's too big to exist, and should be broken up.
Why broken up vs users migrating to a competitive service? Seems like a very simple facet to compete on.
It's hard for users to migrate to a competitive service when there's some form of lock-in, which is usually what happens in practice (often through other services offered by the same company).
This is by far the most ridiculous reasoning I’ve seen for a company being too big. Because too many users get restricted from the service unintentionally then the provider is too big?
Some regions floated right to explanation and right to human review for automated processes. I don't know if any passed, but if they did, it would definitely mean the service has to take it into account.
You have summed it up quite nicely, but I don't see why it's so ridiculous? If the social costs incurred by corporations past a certain size become unacceptable, why shouldn't we, as a society, limit their size? There's no natural right to form an LLC.
Can you please elaborate on bad actors absuing the appeals process? Is your point about how everyone will automatically appeal, making it difficult for genuine queries to receive the human attention they need? Or is there another vector of abuse you were thinking of?
That's basically it.
If every action taken against an account by automation is appealed, then the automation becomes worthless.
In gaming forums that are run by the developer, such as the World of Warcraft or League of Legends forums, I have very frequently seen people whining and complaining that their accounts were banned for no reason until a GM or moderator finally pipes in and posts chat logs of the user spamming racial slurs or some other blatant violation of ToS.
We see that on HN too, where people who have been banned/hellbanned with ample warning are often complaining that it's because "hackernews groupthink" but when you look back at their comment history they call someone some redpilly insult in every comment they've ever made on the website.
1 reply →
It’s even worse than that because the bad actors are doing this at scale and will have automation to auto-appeal while normal people will sometimes shrug and decide it’s not worth it. So your appeals queue likely contains a higher flow of bad actors than the distribution of FPs.
It's interesting to me how Bloom Filters avoid the uncanny valley between probably correct and definitely correct. I don't know if this is a technological difference between problem domains or a purely ideology/mindset.
Dividing a problem by 10 should get notice. By 100 (eg, Bloom Filters) respect. By 1000, accolades. Dividing a problem by infinity should be recognized for what it is: a logic error, not an accomplishment.
Most times when I'm trying to learn someone else's process instead of dictating my own, I'm creating lists of situations where the outcomes are not good. When I have a 'class', I run it up the chain, with a counter-proposal of a different solution, which hopefully becomes the new policy. Usually, that new policy has a probationary period, and then it sticks. Unless it's unpopular, and then it gets stuck in permanent probation. I may have to formally justify my recommendation, repeatedly. In the meantime I have a lot of information queued up waiting for a tweak to the decision tree. We don't seem to be mimicking that model with automated systems, which I think is a huge mistake that is now verging on self-inflicted wound.
Perhaps stated another way, classifying a piece of data should result in many more actions than are visible to the customer, and only a few classifications should result in a fully automated action. The rest should be organizing the data in a way to expedite a human intervention, either by priority or bucket. I could have someone spend tuesday afternoons granting final dispensations on credit card fraud, and every morning looking at threats of legal action (priority and bucket).
Yes; decentralization
that's not a solution to a problem.
end users don't want to run their own spam and moderation filters, and they definitely do want them.