Comment by altcognito
1 day ago
I haven't tried what you guys are talking about myself, but I think when agents begin talking to other agents, the chances for unpredictable goal mixing and confusion resulting in really unexpected and bad behavior is a lot higher.
That being said, the danger is measured in computer damage, which can be a lot personally and to a company, but less existential, so your mileage may vary as to how "scary" it is.
When nearly all money in the world is stored in computers, the danger is quite existential.
This seems traceable back to GIGO. If you don't understand the software you're using, don't use it.
What scares me are the people who think that this software is the equivalent of an perfectly-smart elf in a box and use it blindly.