Comment by ewjordan
8 years ago
A/B testing won’t ever get you a Tesla from a horse.
Right. But if you were already selling Teslas, and along came a smooth-talking product designer with a dream to "improve" it by building a horse instead, A/B testing would make sure that change never saw the light of day.
A/B testing is not a way to get out of having to come up with good ideas yourself. It's a way to validate that your ideas are any good in the first place before betting the whole company on them.
A/B testing is most critical when evaluating big changes to a product, because those are the changes most likely to completely blow up the business. Otherwise it's left up to the opinion of the highest paid person in the room, and people are notoriously bad at guessing how customers will respond to change.
Indeed. When I was advocating for A/B testing at the last company I worked for, I tried hard to teach people that every test should have a sane hypothesis behind it based on some kind of usability theory. Not just “let’s try changing the header font color to red and see what happens!”
I worked for a company with an obsessed AB culture. Biggest hole I found. Clicks != satisfaction. Time spent on site also != satisfaction.
There’s something very powerful just talking to the user, feeling their pain and working with them to fix it until they say “this is awesome”.