Comment by exhaze
1 day ago
Can you show me any longitudinal studies that show examples of a causal connection between incrementality of latency and churn? It’s easy to make such a claim and follow up with “go measure it”. That takes work. There are numerous other things a company may choose to measure instead that are stronger predictors of business impact.
There is probably some connection. Anchoring to 10ms is a bit extreme IMO because it’s indirectly implying that latency is incredibly important which isn’t universally true - each product’s metrics that are predictive of success are much more nuanced and may even have something akin to the set of LLM neurons called “polysemantic” - it may be a combination of several metrics expressed via some nontrivial function that are the best predictor.
For SaaS, if we did want to simplify things and pick just one - usage. That’s the strongest churn signal.
Takeaway: don’t just measure. Be deliberate about what you choose to measure. Measuring everything creates noise and can actually be detrimental.
Human factors has a long history of studying this. I'm 30 years out of school and wouldn't know where to find my notes (and thus references) , but there are places where users will notice 5ms. There are other places where seconds are not noticed.
The web forced people to get used to very long latency and so fail no longer comment on 10+ seconds but the old studies prove they notice them and shorter waits would drive better "feelings". Back in the old days (of 25mhz CPUs!) we had numbers of how long your application could take to do various things before users would become dissatisfied. Most of the time dissatisfied is not something they would blame on the latency even though the lab test proved that was the issue, instead it was a general 'feeling' they would be unable to explain.
There are many many different factors that UI studies used to measure. Lag in the mouse was a big problem, not just the point movement either: if the user clicks you have only so long before it must be obvious that the application saw a click (My laptop fails at this when I click on a link), but didn't have to bring up the respond nearly as fast so long as users could tell it was processing.
https://www.gigaspaces.com/blog/amazon-found-every-100ms-of-...
Here is a study on performance that I did for JavaScript in the browser: https://github.com/prettydiff/wisdom/blob/master/performance...
TLDR; full state restoration of a OS GUI in the browser under 80ms from page request. I was eventually able to get that exact scenario down to 67ms. Not only is the state restoration complete but it covers all interactions and states of the application in a far more durable and complete way than big JavaScript frameworks can provide.
Extreme performance showed me two things:
1. Have good test automation. With a combination of good test automation and types/interfaces on everything you can refactor absolutely massive applications in about 2 hours with almost no risk of breaking anything.
2. Tiny performance improvements mean massive performance gains overall. The difference in behavior is extreme. Imagine pressing a button and what you want is just there before your brain can process screen flicker. This results in a wildly different set of user behaviors than slow software that causes users to wait between interactions.
Then there are downstream consequences to massive performance improvements, the second order consequences. If your software is extremely fast across the board then your test automation can be extremely fast across the board. Again, there is a wildly different set of expectations around quality when you can run end-to-end testing across 300 scenarios in under 8 seconds as compared to waiting 30 minutes to fully validate software quality. In the later case nobody runs the tests until they are forced to as some sort of CI step and even then people will debate if a given change is worth the effort. When testing takes less than 8 seconds everybody and their dog, including the completely non-technical people, runs the tests dozens of times a day.
I wrote my study of performance just a few months before being laid off from JavaScript land. Now, I will never go back for less than half a million in salary per year. I got tired of people repeating the same mistakes over and over. God forbid you know what the answer is to cure world hunger and bring in world peace, because any suggestion to make things better is ALWAYS met with hostility if it challenges a developer's comfort bubble. So, now I do something else where I can make just as much money without all the stupidity.