The burden you are placing is too high here. Do you demand controlled trials for everything you do or else you refuse to use it or accept that other people might see productivity gains? Do you demand studies showing that static typing is productive? Syntax highlighting? IDEs or Vim? Unit testing? Whatever language you use?
Obviously not? It would be absurd to walk into a thread about Rust and say “Rust doesn’t increase your productivity and unless you can produce a study proving it does then your own personal anecdotes are worthless.”
Why the increased demand for rigor when it comes to AI specifically?
Typically I hear how other people are doing things and I test it out for myself. Just like I'm doing with AI
Actually IDEs vs vim are a perfect analogy because they both have the ability to feel like they're helping a tonne, and at the end of the work day neither group outperforms the other
I'm not standing on the sidelines criticizing this stuff. I'm using it. I'm growing more and more skeptical because it's not noticably helping me deliver features faster
At this point I'm at "okay record a video and show me these 3x gains you're seeing because I'm not experiencing the same thing"
The increased demand for rigor is because my experience isn't matching what others say
I can see a 25% bump in productivity being realistic if I learn where it works well. There are people claiming 3-10x. It sounds ridiculous
Do you just believe everything everybody says? No quantifiable data required, as long as someone somewhere says it it must be true?
One of the reasons software is in decline is because it's all vibes, nobody has much interest in conducting research to find anything out. It doesn't have to be some double blinded peer reviewed meta analysis, the bar can still be low, it just should be higher than "I feel like"...
I’m not sure why you’d need or want a randomised controlled trial to determine the colour of the sky. There have been empirical studies done to determine the colour and the reasoning for it - https://acp.copernicus.org/articles/23/14829/2023/acp-23-148... is an interesting read.
If you point a spectrometer at the sky during the day in non-cloudy conditions you will observe readings peaking in the roughly 450-495 nanometers range, which crazily enough, is the definition of the colour blue [0]!
Then you can research Rayleigh scattering, of which consists of a large body of academic research not just confirming that the sky is blue, but also why.
But hey, if you want to claim the sky is red because you feel like it is, go ahead. Most people won't take you seriously just like they don't take similar claims about AI seriously.
pretending the only way anybody comes to a conclusion about anything is by reading peer-journals is an absurdly myopic view of epistemological practices in the real world
I don't get it? Yes you should require a valid reason before believing something
The only objective measures I've seen people attempt to take have at best shown no productivity loss:
https://substack.com/home/post/p-172538377
https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
This matches my own experience using agents, although I'm actually secretly optimistic about learning to use it well
The burden you are placing is too high here. Do you demand controlled trials for everything you do or else you refuse to use it or accept that other people might see productivity gains? Do you demand studies showing that static typing is productive? Syntax highlighting? IDEs or Vim? Unit testing? Whatever language you use?
Obviously not? It would be absurd to walk into a thread about Rust and say “Rust doesn’t increase your productivity and unless you can produce a study proving it does then your own personal anecdotes are worthless.”
Why the increased demand for rigor when it comes to AI specifically?
Typically I hear how other people are doing things and I test it out for myself. Just like I'm doing with AI
Actually IDEs vs vim are a perfect analogy because they both have the ability to feel like they're helping a tonne, and at the end of the work day neither group outperforms the other
I'm not standing on the sidelines criticizing this stuff. I'm using it. I'm growing more and more skeptical because it's not noticably helping me deliver features faster
At this point I'm at "okay record a video and show me these 3x gains you're seeing because I'm not experiencing the same thing"
The increased demand for rigor is because my experience isn't matching what others say
I can see a 25% bump in productivity being realistic if I learn where it works well. There are people claiming 3-10x. It sounds ridiculous
4 replies →
I honestly wish we had studies that truly answered these Qs. Modern programming has been a cargo cult for a good 20 years now.
1 reply →
Do you just believe everything everybody says? No quantifiable data required, as long as someone somewhere says it it must be true?
One of the reasons software is in decline is because it's all vibes, nobody has much interest in conducting research to find anything out. It doesn't have to be some double blinded peer reviewed meta analysis, the bar can still be low, it just should be higher than "I feel like"...
8 replies →
Why do you believe that the sky is blue? What randomized trial with proper statistical controls has shown this to be true?
I’m not sure why you’d need or want a randomised controlled trial to determine the colour of the sky. There have been empirical studies done to determine the colour and the reasoning for it - https://acp.copernicus.org/articles/23/14829/2023/acp-23-148... is an interesting read.
I can see it, it's independently verifiable by others, and it's measurable
6 replies →
If you point a spectrometer at the sky during the day in non-cloudy conditions you will observe readings peaking in the roughly 450-495 nanometers range, which crazily enough, is the definition of the colour blue [0]!
Then you can research Rayleigh scattering, of which consists of a large body of academic research not just confirming that the sky is blue, but also why.
But hey, if you want to claim the sky is red because you feel like it is, go ahead. Most people won't take you seriously just like they don't take similar claims about AI seriously.
[0] https://scied.ucar.edu/image/wavelength-blue-and-red-light-i...
2 replies →
[flagged]
pretending the only way anybody comes to a conclusion about anything is by reading peer-journals is an absurdly myopic view of epistemological practices in the real world
Nobody is pretending that's the case...
4 replies →
[flagged]