Comment by bryanhogan
10 hours ago
This is very close to what you call usability testing in the field of UX design.
The Nielson Norman Group has a good introduction to this: https://www.nngroup.com/articles/usability-testing-101/
Is this interesting to people on HN?
I majored in a mix between coding and design.
I think I read somewhere (and in any case it matches my experience) that the first user will catch 50% of the problems and that with 5 users you can catch close to 90%. So human testing is not just invaluable, it's pretty cheap for the benefits it brings.
At every UX job I've had we generally ran usability tests with 6 participants. This was before the online usability testing tools proliferated and we had a lab in our office (one way mirror with observation room and everything). Every UX researcher I worked with said based on the available research and personal experience it is as you said: more than 5-6 participants would hit diminishing returns pretty hard and wasn't really worth it.
Probably you learned that from them! The first user is 31%, from their testing.
https://www.nngroup.com/articles/why-you-only-need-to-test-w...
https://www.nngroup.com/articles/how-many-test-users/
Yes, very probably, thanks!
Its not just QA and bug hunting.
You want to hire your target audience, which may be vastly different from yourself, and suddenly you discover that the language is throwing them off, the color scheme brings different meanings, and they just don't understand the flow which felt completely natural for you.
It's more like the first user catches all of the critical bugs and a handful of users catch all of the high impact bugs. Cause that's kinda the definition of the impact of critical and high impact bugs.
I was also going to post "this sounds like usability/user research".
But I guess we all re-discover things when we need to.
The biggest issue I've seen with tech docs is often expert users of a given tool or product are enlisted to write the docs, because of their expertise.
But then ironically they end up writing those docs for an audience that shares their level of expertise, rather than for the intended audience.
So you end up with lots of assumptions or leaps of logic in the docs that the intended audience can't follow.
There's the famous quip that Steve Jobs hated focus groups. What people miss is that he (and Apple overall) was heavy on UX testing.
I'd wager this is interesting, seems there are a lot of users here building things on the side (or main projects) that need clear UX if they're anything like me, they don't have that UX background to know the best practices. I know I just clicked your link anyway...
Yes very much so and thanks for the link.