Comment by godwinson__4-8

It's a good objection, and one I was waiting for. Thank you for raising it.

> Do you have a response to the threat of cognitive surrender and cognitive debt as a second order effect from heavy LLM use and how that could undermine humanity’s ability to steer these correlation engines toward your preferred hypothetical outcomes without AGI?

Yes, by way of analogy. Primordial man was more familiar with murder and the slaying of wild animals for sustenance. To lament this loss is to be something like those men I mentioned earlier, striving to be screamed at while shivering in the cold in some pantomime of ancient virtue.

The world is complicated. Humanity cannot continue to advance without finding a way to offload such complexity to technology. This has always been the way, and on balance, we are better off for it. The problems of the future are likely too complicated, along too many dimensions, for human cognition to solve alone. In many ways, humans have always known this. It's where the notion of God comes from.

I'm also not sure what sort of cognition will be needed to "steer". It's a good question. This is an interesting problem, but I think the way you frame it rests on a few faulty assumptions.

1. That humans, absent of LLMs, even "steer" their own societies. The average human has really no say in affairs of even the most liberal modern states, certainly not in its underpinnings. A lucky few get to frame the terms of constitutions or revolutions or polities. Most who are cited to have died in such efforts were not the ones to actually make the choices we still live with today. That power has always been the province of a privileged few. This is an inescapable side effect of complexity. In this way, LLMs are merely a continuum of historical trends, and as such, are largely unavoidable.

2. That humans will be the necessary or only interface to steer LLMs. It is far more likely some engineering around adversarial machines or consensus of other automation will have to "enforce" certain types of steering, irrespective of the wishes of any particular human agent. This is likely desirable, in the same way most modern states do not wish to be hijacked by executive fiat.

3. That you need broad buy in to steer LLMs. Again, historically this has never been the case. Revolutions are typically won by a minority, most of the population in any conflict usually remains unaligned and simply wants to stay out of harms way. Most people do not want to steer anything, they just want to survive and be left alone. Government is not a luxury - it is insurance against being murdered in your own bed. Most people intuitively understand this, even if they do not understand or appreciate the genesis of their own particular states.

But I agree, it is a good question, and I am not convinced we will achieve that "love letter" state. That is simply one of the few positive outcomes I envision. But I personally find it more likely humans will destroy each other before such a state could be meaningfully achieved at scale. I am just trying to be more optimistic.