Comment by WarmWash
10 hours ago
A bit of caution with this line of thinking, because I was blindsided by it after more than decade of hand waving away "academics"
When you have the mind of "the models are wrong", you tend to be brought there by select counter examples, and in that space you only feed on counter examples. "The experts are wrong (and you can be right in 15 minutes if you consume this media!)" is abundant.
The actual reality is "the models work great until they don't". Not "the models don't work."
Ironically, the experts, the ones building the "wrong" models, tend to be the ones most aware of this.
> Ironically, the experts, the ones building the "wrong" models, tend to be the ones most aware of this.
This varies substantially by field, or by subfield. There is a vast decades-long intellectual wasteland of bad economics predicated on bad models that are only appealing on normative grounds. The entire field of behavioral psychology might be a scam. Etc etc. Science advances one funeral at a time hard part because people are unwilling to let go of their models, even when they have outlived their usefulness or have been simply proven too wrong to be useful even in their original purpose.
> The entire field of behavioral psychology might be a scam
I studied cognition and came to the very same, sad conclusion. There’s a reason there was a reproduction crisis and it’s far from over. I remember vividly how one of my professors, during a paper debate, would end criticism with: “but it’s been published” as if that handwavy gesture could explain away serious research gaps that even graduate students saw through right away. Shoddy statistics in a system that rewards quantity can do a lot of damage.
Until this moment, I hadn't even thought that economics might be a scientific field. Do they really see themselves as one? It hasn't seemed like one from the outside.
without getting too deep into "field purity" [0], it's certainly a field of study insofar as game theory, psychology, anthropology, etc. are fields of study. It's possible to make certain models of behavior or outcome potentials, but running experiments is significantly more difficult than e.g. physics or biology.
So it's science inasmuch as anything with n=1 repeatability can be science
[0]: https://xkcd.com/435/
1 reply →
Economics is a scientific field in the same sense that Astrology and Politics are. You formulate theories about what your customers want to hear. Then test those theories by telling your customers things, while observing both immediate reactions and how situations develop over time. That latter in conjunction with formulating updated theories about what your customers want... Endless loop.
Do they believe it's a science? Assume that Upton Sinclair's famous quote applies.
4 replies →
It's because the models have different purposes in decision making, in ways which decision makers don't often appreciate.
Models are still great at helping us understand the world and are in many ways the best thing we have. The problem is that today we are overly relying on them to make real policy interventions on an ecology (e.g. what is a "healthy" amount of animals to cull or fish) based on an ecosystem, which is just a poor model.
If we could get rid of this idea of "stability" and "equilibrium" in economics and ecology, I would be a happy man.
> Ironically, the experts, the ones building the "wrong" models, tend to be the ones most aware of this
Exactly why I'm here :-)
Decision making is risk management, and is often simpler than making predictions. You need to think in terms of utility functions and you need to stay far away from the catastrophic regions. You often don't need super accurate predictions for that.
One example is pandemics. The thing is pandemics are extremely fat tailed in terms of fatalities. You need a lot of data to fit any outbreak model and no one has time for that. But the reaction to diseases out breaks is very simple. Is it super deadly and contagious? If yes, shut everything down! This is essentially what Asian countries like Vietnam and Taiwan did during Covid and they handled it way better than the West. I remember arguing about outbreak models, which is a completely useless activity.
Maybe the best way to illustrate our point is to use weather models, as it's pretty tangible for regular people.
Weather models are ridiculously advanced running on super computers crunching a massive global network of real time data. But they still aren't anywhere close to perfect.
That being said, just because it rains on your birthday when it was said to be sunny, doesn't mean you delete your weather app, call meteorology pseudoscience, and start a substack of "the forecast was wrong again" blog posts.
People intuitively grasp this foolishness because they constantly interact with weather models. But for things they have almost no contact with, it's easy to write it off on a single "bad forecast"
(I'll also admit the caveat that not every model is as good (or bad) as weather models, another dimension at play to throw a wrench in peoples gears)
The weather is easy to measure though, which leads to a continuing improvement of the models. In a lot of fields it’s very difficult to make experiments and measure the results. In those fields models can stay very bad for a long time.
I love this analogy... The Heresy of Meteorology! Everything You Thought You Knew about Weather is Wrong. How Weather Reports are Sapping the Joy From Your Life. The Truth about Weather! The Misguided Science of Predicting Mother Nature.
2 replies →
> When you have the mind of "the models are wrong", you tend to be brought there by select counter examples, and in that space you only feed on counter examples.
Box's (possibly apocryphal) aphorism, "all models are wrong, but some are useful", is the better mindset.
Obviously we can never perfectly model nature, but we can often get close enough to form useful predictions. A minor predictive failure does not necessarily mean you throw the model, and all of its predictions, out entirely.
> "The experts are wrong (and you can be right in 15 minutes if you consume this media!)" is abundant.
Social media, sadly, preys on the feeble-minded or willingly-deceived :(
the unspoken permanent wrench in the gears lies in chaos theory. models always have some level of consolidation of reality that gets treated in blocks. While this has intuitive inaccuracies like rounding errors, edge cases, and such, we also know reality often shows emergent behavior from chaotic interactions in a way only understandable after it's actually happened. these could be erorrs that could completely break the models relevance
After observing the emergent behavior, we can often then incorporate it into the model by breaking up prior consolidations into more parts, but the fundamental problem still remains. This makes models for things that arent part of a relatively rapid feedback loop for model improvement (like climate change) to be very vulnerable unknowns. thats not to say they are completely useless, but it can very much take away the weight behind any specificity of the models results, and should open up a second conversation about the direction in which the model might be expected to fail.
Additionally, when creating models that involve human behavior you can appeal to game theory, psychology, sociology, statistics, etc but ultimately chaos will be in full force. now with AI it is going to become a factor amongst automation as well in a way it previously was not. there is no practical way to model how the weights of neural networks might act unexpectedly in various contexts.
all this to say models are not often that great at giving useful predictions as much as they are great at building foundations of understanding of relationships in complex/complicated and what "clean" situations might look like, which can then be taken into consideration of what reality is likely to look like and happen. this distinction is important because currently, automation tools and AI lack that final chaotic adjustment that astute humans are able to apply. Ai has gotten very good at making complicated models but it ultimately is, by design imo whether intentional or not, limited in the same way models themselves are.
that final adjustment to make real world decisions and have personal accountability is more rooted in beliefs/feelings than it is in model outputs, albeit the model outputs help to refine it.
Same thing said by Isaac Asimov in 1986: "The Relativity of Wrong"
https://web.williams.edu/Mathematics/sjmiller/public_html/23...
I was always a fan of what a former senior colleague ingrained in me early in my career:
A model that has no predictive utility is a bad model, but every model hits its limits in terms of predictive power. Some models have a very steep cliff in accuracy at the "edges", and some gradually lose fidelity like an out-of-focus photograph.
Most anti-science criticism I see is completely ignorant of what it really means to model a system or phenomenon: the challenges, the limitations, the end goal, the criteria for success, etc. It's sort of a statistical / scientific ignorance that's deeply cross-discipline. To perform science in a lot of ways is to engage with what it means to build a model, and in some ways, interrogate the universe.
If you're both an idiot and overconfident, it's tempting to throw your hands up and not engage with something as complex as "modeling", and just decide almost out of whimsy what the universe should truly be. But at least some people who find modeling too intellectually taxing at least admit that there are things that they'll never understand.