Comment by godelski

  > successfully doing it is another

Not to mention successfully measuring success or failure. The overwhelming majority of people I know (from your average Joe to your tenured professor at elite universities, from philosophers to physicists[0]) underestimate the difficulty of measuring things.

Almost everyone treats any metric as they would a ruler or tape measure. Even your standard ruler is not as good of a measuring device as you probably think! But this becomes a huge mess when we start talking about any measurement of statistics or some other abstraction. People treat metrics and algorithms as black boxes, rather than tools. Tools still require craftsmen, who understands: when they work, when they don't work, when they can be used in a pickle, what can be substituted in a pickle, their limits, what new problems they create, and so on. It is incredible how much complexity there is to things that appear so simple. But then again, that's why you get things like an engineering manual on o-rings that is over a thousand pages. And even those aren't comprehensive.

I'm not suggesting we all need to be "master craftsmen", but I actually think we would all do better if we recognized that everything has more depth than might appear. If only to give people a moment of pause to question if they are actually doing things the right way. There's always a better way. The real trick is learning what's good enough and you'll never know what is good enough when everything is simple.

[0] The exception tends to be those that need to work with high precision, since with these jobs you tend to be forced to deal with this in an explicit manner. So more common among people like machinists or experimental physicists. Though sometimes this ends up worse as they can end up operating on vibes. I think it happens when intuitions are successful for too long and not enough meta-analysis is done to update them.