Comment by js8
6 years ago
Indeed, I suspect there is a curse of expertise. If you're an expert, you begin seeing everything as so nuanced, as so fragile, that you will dismiss lots of radical innovation as too simplistic. (And I think especially it affects motivation to actually try something new.)
But if somebody somewhat naive comes along, and just pushes through with sheer effort, they might succeed where many experts have predicted a failure.
Or more likely, the newbies will solve the “unsolvable” problem by removing some of the constraints that the old guard was holding inviolable. The newbies crow about their success for a few years while everyone struggles to work around the constraint violations.
NoSQL is the biggest example of this. Ignore everything we learned about ACID and just use key-value stores with no transactions or relations. People can build their own if they need them, right? And duplicating data to work around missing relationships is not a problem because storage is cheap? Then we get an explosion of new databases, each of which solve a subset of the missing functionally problems, with varying degrees of success.
I suspect this came across as more snide than I meant it. Sometimes holding a treasured constraint is the wrong thing and the old guard of experts failed to understand that not every business problem needed the full solution. But it annoys me for some reason the attitude of “we just solved this problem that experts could not solve for decades” when the nuance is that only a subset of the problem was solved, with potentially extraordinary effort required to re-introduce those missing constraints.
I can't quite buy your example, because NoSQL is pretty much a reinvention of VSAM and IMS DB (not IMS DC).
Regardless, I don't think it's necessarily that the expert would not want to drop some design constraint. It's more likely that it is genuinely hard to decide, which of them to drop and which of them to keep, because there are so many and problems are complex.
But if you're somewhat new (not a beginner either), you don't see all these, and you can benefit from the ignorance. Unfortunately, I think it cuts both ways, it's far more likely that ignorance will actually hurt you. But for a small number of people, ignorance can lead to lucky innovation.
I call that the Procrustean Bed: Simplify the solution by cutting pieces off the problem.
There doesn't even need to be great effort. Sometimes the experts are just so good at making every idea look like shit that they don't even try.