Comment by TipVFL
8 years ago
They've probably done testing and found that for some significant percentage of married couples, seeing their spouse's posts caused them to reduce their Facebook usage. So unhappy marriages are ruining it for everyone.
I feel like the general trend of UI simplification and a/b testing everything has companies chasing after anything that will push their general metrics up in the short term, at the expense of slowly disenfranchising a large portion of their users.
I interviewed at Netflix recently for a position on their UI team, and I questioned the interviewer about a lot of annoying aspects of the Netflix interface. He was aware of every complaint, and his explanation for every one of them was he wishes he could change it but the a/b testing shows they get higher engagement with it the way it is. So, for example, they show you movies you have already said you don't like, because enough people give a thumbs down and then later watch the movie. In the short term very few people are going to cancel Netflix because of this, but enough of these decisions in succession is erosive to customer goodwill.
Personally, these kind of decisions wouldn't bother me that much if apps and websites made it easy to customize your experience. Facebook makes it difficult by opaquely shaping your newsfeed and giving you what seems like hundreds of options across a half dozen settings pages. Netflix makes it difficult by just giving you practically no options, but there's enough demand that's there a tiny industry in just making websites and plugins to make Netflix usable.
> In the short term very few people are going to cancel Netflix because of this, but enough of these decisions in succession is erosive to customer goodwill.
The “auto preview” is so unpleasant that I’ve essentially stopped using Netflix entirely. I’ve no doubt that their testing shows it increases engagement, but I find it so jarring and abrasive that I rarely open the app any more. I’m sure they still have content I want to watch, but I can’t stand the experience of trying to find it.
I would love to have more insight into their A/B engagement metrics. I’m sure they capture if immediate engagement goes up when a feature is enabled, and probably the same over time periods of days or weeks. Do they also capture if total app usage declines 75% over 6 months after a feature is enabled?
I agree. You can't even leave Netflix running on a device and leave the room because it will just start playing something without any actual user input! And it appears none of the apps let you turn this behavior off.
Netflix already seems to have no problem getting people to watch hours and hours of shows as a result of having high quality content. Why do they need to perform cheap tricks like auto-preview to bump up user stats, as others have pointed out, at the expense of user happiness/preference?
That’s funny - I Like the auto preview, in fact I would like them to go further and advertise their content to me, as live TV does. When I watch Netflix at the end of a hard day I’m in lazy consumer mode, I want content to be suggested to me. I hate wading through static dull pictures which give me very few clues as to the content they represent. Netflix are unstoppable - they offer incredible value compared to extortionate SKY, i cancelled their £120 per month package 6 months ago and I have never regretted it.
I'm listening to music or half-watching twitch whilst scrolling through Netflix to decide what I'll watch with supper. It's incredibly frustrating to have them autoplay video content, to the extent that I've started going back to browsing torrent sites instead.
It doesn't help that their website is a laggy piece of shit with miserable discover-ability. But, you know. Ugh.
2 replies →
I also like auto preview. But I would never want them to start advertising their shows to me (like Amazon Prime does), that's going too far.
> The “auto preview” is so unpleasant that I’ve essentially stopped using Netflix entirely.
If you find a lower powered device such as the previous generation of Amazon Fire TV, it does not auto-preview or auto-run trailers at the top of the screen - instead you just get a static image of the show.
Now if only I could get it to do that on my PS4 :(
I'm definitely not planning to buy a dedicated crappy device for Netflix, though. :/ I want fewer devices. I've only still got an Xbox One for Blu-Ray/DVD and HBO Now, which inexplicably isn't available for my TV.
I guess A/B testing will tell you if the horse will become faster if you change from iron shoes to carbon fiber ones.
A/B testing won’t ever get you a Tesla from a horse.
A/B testing won’t ever get you a Tesla from a horse.
Right. But if you were already selling Teslas, and along came a smooth-talking product designer with a dream to "improve" it by building a horse instead, A/B testing would make sure that change never saw the light of day.
A/B testing is not a way to get out of having to come up with good ideas yourself. It's a way to validate that your ideas are any good in the first place before betting the whole company on them.
A/B testing is most critical when evaluating big changes to a product, because those are the changes most likely to completely blow up the business. Otherwise it's left up to the opinion of the highest paid person in the room, and people are notoriously bad at guessing how customers will respond to change.
Indeed. When I was advocating for A/B testing at the last company I worked for, I tried hard to teach people that every test should have a sane hypothesis behind it based on some kind of usability theory. Not just “let’s try changing the header font color to red and see what happens!”
1 reply →
It will also tell you that pumping horses full of steroids will make them go faster. Tech's absurd obsession with metrics and statistics will be their undoing.
If you do enough binary switching, you eventually will get a Tesla. Assuming a Tesla is optimal. Let’s test to find out.
How do you think the eyeball arose from unicellular eukaryotes?
I don't think so. A/B testing is like gradient descent which is a greedy algorithm. You move in the direction that locally looks best. Evolution on the other hand allows for suboptimal species to persist for enough time to let them develop their advantage. (In the language of optimization evolution allows you to go past the local optimal and reach global optima by allowing you to move in non-optimal direction -- as long as the move is not catastrophic.)
4 replies →
You won't get out of local minima/maxima with hill climbing though.
Eyeballs are nowhere near optimal either.
local maxima trap hill climbing style optimizing, many (most?) times improvements require jumping large sub optimal chasms that can never be crossed by gradual improvement.
Says every product person I've ever worked with who didn't want to actually be data driven.
I understand the fear, though - when you're data driven, you can actually numerically measure the contribution of every feature that's tested, and in a lot of cases figure out the exact impact on revenue. In a culture where everything is tested, it's a very small step from there to stack ranking your product designers based mostly on how the features they designed did.
I'm not necessarily against that as a valid way to measure job performance if it's done intelligently (for one, realizing that there's a lot of blind luck and variance, and it takes time to smooth out) - I mean, if you're in sales and you're not booking any sales, you don't get to hide from that. But it's also really easy to get ranking and evaluation schemes wrong, so I understand why people would be nervous about it and prefer soft-skill-based evaluations instead.
1 reply →
There are two kinds of product people: those who can't design and those who don't understand statistics.
> he wishes he could change it but the a/b testing shows they get higher engagement with it the way it is
The upshot of this is that if Netflix or Facebook _do_ see a slow drift of users away from the platform, they are institutionally incapable of understanding why. Engagement is not the same thing as satisfaction, which is far harder to measure.
(I'm also now wondering how static the results of A/B testing really are - if an idea fails AB even once does the company abandon it forever?)
But engagement is what advertisers pay for, isn't it ? Clicks ! Clicks ! Clicks !
So I'm pretty sure that they are institutionally incapable of understanding why for a very good reason.
yes, a/b testing optimizes for an average user that dosent exist, and everyone is unhappy with.
Reminds me of the old tale of the man who drowned in a lake with an average depth of 3 feet.
A/B doesn’t design for an average user; it can be all, a tiny niche, or something in between depending on what you’re measuring. For example, DAU and engagement will be affected by drastically different distributions of users.
Did you ask what their definition of engagement is, and why it's the primary measure?