Comment by randomImmigrant

Can someone with direct AI experience respond:

Does RSI necessarily need to be pointed towards “AGI”, whatever that is?

Or can you have a recursive self improvement loop where the objective is to create models that are more legible to humans? Or better at attributing the source text if it’s meaningfully similar to output? Or architectures for models that are increasingly better at human-AI cowork rather than automation?

From my own understanding, nothing at all says the frontier is defined by the quest for “AGI”. This definition of the frontier assumes human intelligence itself has peaked and will remain stable, so how well defined can this goal ever be, if AGI stands in comparison to human intelligence for its definition?

To me, Amodei’s writing just reeks of posturing. Genuine action driven by this fear they claim to have would be meaningful.

Even setting aside moral quandaries, isn’t it basic project process to have your company’s internal goals be tethered to what people want, rather than what you calculate is inevitable?

There are genuinely other frontiers to explore, and Anthropic would do a lot to mitigate the current slide if it decided to put its resources towards another direction for AI.

Take back the agency that you are so blithely surrendering to models you do not understand. There is no inevitability to this path. There are critical, meaningful choices, and Anthropic wouldn’t be violating capitalism by taking an alternate that is more tethered to what users want and need. Maybe by asking them first, at scale.