Comment by nojvek
8 years ago
I guess A/B testing will tell you if the horse will become faster if you change from iron shoes to carbon fiber ones.
A/B testing won’t ever get you a Tesla from a horse.
8 years ago
I guess A/B testing will tell you if the horse will become faster if you change from iron shoes to carbon fiber ones.
A/B testing won’t ever get you a Tesla from a horse.
A/B testing won’t ever get you a Tesla from a horse.
Right. But if you were already selling Teslas, and along came a smooth-talking product designer with a dream to "improve" it by building a horse instead, A/B testing would make sure that change never saw the light of day.
A/B testing is not a way to get out of having to come up with good ideas yourself. It's a way to validate that your ideas are any good in the first place before betting the whole company on them.
A/B testing is most critical when evaluating big changes to a product, because those are the changes most likely to completely blow up the business. Otherwise it's left up to the opinion of the highest paid person in the room, and people are notoriously bad at guessing how customers will respond to change.
Indeed. When I was advocating for A/B testing at the last company I worked for, I tried hard to teach people that every test should have a sane hypothesis behind it based on some kind of usability theory. Not just “let’s try changing the header font color to red and see what happens!”
I worked for a company with an obsessed AB culture. Biggest hole I found. Clicks != satisfaction. Time spent on site also != satisfaction.
There’s something very powerful just talking to the user, feeling their pain and working with them to fix it until they say “this is awesome”.
It will also tell you that pumping horses full of steroids will make them go faster. Tech's absurd obsession with metrics and statistics will be their undoing.
If you do enough binary switching, you eventually will get a Tesla. Assuming a Tesla is optimal. Let’s test to find out.
How do you think the eyeball arose from unicellular eukaryotes?
I don't think so. A/B testing is like gradient descent which is a greedy algorithm. You move in the direction that locally looks best. Evolution on the other hand allows for suboptimal species to persist for enough time to let them develop their advantage. (In the language of optimization evolution allows you to go past the local optimal and reach global optima by allowing you to move in non-optimal direction -- as long as the move is not catastrophic.)
Nope. A/B testing depends on what you choose to mutate. The problem is that we humans intentionally change the things. Nature randomly changes the things.
You’re not going to get to a global optimum driven by human choice of what to test (only local optima at best) unless the human setting up the tests is some sort of sage.
3 replies →
You won't get out of local minima/maxima with hill climbing though.
Eyeballs are nowhere near optimal either.
local maxima trap hill climbing style optimizing, many (most?) times improvements require jumping large sub optimal chasms that can never be crossed by gradual improvement.
Says every product person I've ever worked with who didn't want to actually be data driven.
I understand the fear, though - when you're data driven, you can actually numerically measure the contribution of every feature that's tested, and in a lot of cases figure out the exact impact on revenue. In a culture where everything is tested, it's a very small step from there to stack ranking your product designers based mostly on how the features they designed did.
I'm not necessarily against that as a valid way to measure job performance if it's done intelligently (for one, realizing that there's a lot of blind luck and variance, and it takes time to smooth out) - I mean, if you're in sales and you're not booking any sales, you don't get to hide from that. But it's also really easy to get ranking and evaluation schemes wrong, so I understand why people would be nervous about it and prefer soft-skill-based evaluations instead.
valid point. However I also often see "politics optimized sales" where the current pet project of whoever has the most swing goes up front even if it's a terrible product and death to revenue.
There are two kinds of product people: those who can't design and those who don't understand statistics.