Comment by Majromax
2 years ago
> I'm curious: what is the rationale for "in an incognito tab" being part of the test harness?
It's the closest we can easily get to the 'average user experience'. Someone who has a long account/cookie history with Google has plausibly trained the site to return more relevant results through implicit user-curation of avoiding obvious-to-them SEO-spam on other queries.
If we posit that every user eventually trains Google to avoid SEO spam, then this begs the question of why Google(/Bing) don't eliminate the SEO spam in the first place.
Besides that, it's not obvious why search engine personalization should dramatically change the basic utility of search results. We should expect personalization to mostly address ambiguities: is 'the best way to set up tables' asking about furniture assembly/carpentry or SQL? None of the author's queries for this article supported such ambiguities, and besides that the results returned (see the final appendix) aren't[†] valid answers to a different interpretation of the question.
[†] -- I think I'd quibble about the 'adblock' question, since a reasonable person might still find an adblocker that works but participates in the 'acceptable ads program' to be sufficient.
> It's the closest we can easily get to the 'average user experience'.
Maybe it's the closest we can get (though I doubt it), but it definitely isn't close enough to tell us anything about the "average user experience".
The average user has been using google for years, without taking any steps to avoid personalization. An incognito session (on a browser / machine / network that is probably fingerprinted...) is pretty much the opposite of that typical usage pattern.
I recognize that just writing a blog post or comment on HN is not a research project so needs to do something quick, but I think it mostly invalidates the experiment. What would get closer would be to devise a few user personas and attempt to search and browse for awhile within those personas before trying the experiment. Or much better yet, put together a focus group comprised of real people within the personas you're interested in, and run the experiment using their real accounts.
> If we posit that every user eventually trains Google to avoid SEO spam
I don't think it's that, I think it's that every user trains it to return results more likely to improve the metric of "more likely to click one of the links", and I think that makes it more, not less, likely that they see what most of us here consider to be spam.
But I don't know! Maybe that's not what this experimental setup would show. But it would be a lot more enlightening than a setup using a fresh incognito window, which reflects the usage pattern of a proportion of search queries that is a tiny rounding error above zero.
Why are you assuming all users are logged in to google all the time?
Because it is objectively the case that the "average user" of the internet has a google cookie in their browser. It doesn't require that they be logged in - though I believe it's likely also the case that the "average user" is indeed logged into a google account - it just requires that they use google search without turning off cookies or specifically blocking google's. Essentially everybody uses google search and essentially nobody cares enough (or would know how) to turn off cookies or block google's cookie.
If this doesn't describe most people you know, you're in a very small bubble. (I'm somewhat in that bubble too, but I still have lots of family and friends who use the internet the normal way.)
Google has billions of user accounts ....
> It's the closest we can easily get to the 'average user experience'
You wouldn’t be really taking the average here though would you? You would be capture the experience someone might have if they were in incognito, using google for the very first time, or using google on another device for the every first time, but not the “average experience”.