Comment by wpietri
8 years ago
One of the things that makes me a little bonkers is when people A/B test, find something that works well for, say, 50% of the people, and then dust off their hands and call it done. Because 50% is a lot of people, so it must be good.
I'd rather that they ask, "Hey, is there something that would be good for the other half of the users?" Software being infinitely soft, we can often find ways to serve both groups.
If Pinterest were a person, we'd think it an asshole, always pulling a bait-and-switch and asking us to create an account in the same brusque way. How much would it really cost them to be more neighborly?
I had friends in Groupon who used this technique. They claimed that forced sign ups didn't really result in a high enough bounce rate, compared to the number of sign ups they got.
Data driven does a poor job of capturing user annoyance. Someone will tolerate being screwed a few times to get a discount. But after a while, it builds momentum.
Though in many cases, startups don't care. They just want to sell off their numbers and exit.
50% is overselling the success rate of login-walls from Google Images by a lot - probably multiple orders of magnitude. And I think you're significantly misrepresenting the A/B test's goal. It's not to find what works best for the most people, it's to find what works best for the company running the test.
If a login wall causes an account creation from 0.5% of people presented with the bait-and-switch, that's not compared to a 99.5% failure rate. It's contrasted with an account creation from 0.1% of people presented with the image they wanted with a "Click here to create an account" prompt in the corner, and it's five times as effective as that alternative.
The same story gets repeated here over and over again regarding the email newsletter sign-up form that steals focus halfway through a blog post. Yes, they increase your traceable readership more than anything else you could put in your blog (again, counting 1% vs 0.1%) . But there is a large percentage of the population who will close your blog permanently if you use them.
Yes, I made up an example for simplicity's sake. And my exact critique is that they don't work best for the company in the long term, because the people using them are often obsessively focused on the short term metrics.
The bait and switch may get them 0.5% signups instead of 0.1% for that interaction. But rarely do people ask, "What distinguishes the people who signed up? What effect are we having on the people who don't sign up? Can we serve those people as well?"
As you say, the long-term effect is pernicious.